[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:33:17 dist-test-slave-rrj5 krb5kdc[17020](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:33:17 dist-test-slave-rrj5 krb5kdc[17020](info): set up 2 sockets
Sep 28 06:33:17 dist-test-slave-rrj5 krb5kdc[17020](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 06:33:18.094332 17013 mini_kdc.cc:121] Time spent starting KDC: real 1.131s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Sep 28 06:33:18 dist-test-slave-rrj5 krb5kdc[17020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:33:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:33:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.157.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab.
Entry for principal kudu/127.16.157.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab.
I20260928 06:33:18.229143 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:42551
--webserver_interface=127.16.157.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=kudu/127.16.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:45483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.157.126:42551
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:18.313884 17033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:18.314073 17033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:18.314095 17033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:18.315793 17033 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:33:18.315837 17033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:18.315852 17033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:18.315865 17033 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:33:18.315877 17033 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:33:18.317301 17033 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:45483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.157.126:42551
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=kudu/127.16.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:42551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:18.317543 17033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:18.317808 17033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:18.323450 17039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:18.323472 17038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:18.323776 17033 server_base.cc:1034] running on GCE node
W20260928 06:33:18.323508 17041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:18.324244 17033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:18.324568 17033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:18.325760 17033 hybrid_clock.cc:648] HybridClock initialized: now 1790577198325724 us; error 30 us; skew 500 ppm
Sep 28 06:33:18 dist-test-slave-rrj5 krb5kdc[17020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577198, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:18.328922 17033 init.cc:377] Logged in from keytab as kudu/127.16.157.126@KRBTEST.COM (short username kudu)
I20260928 06:33:18.329468 17033 webserver.cc:460] Webserver started at http://127.16.157.126:45693/ using document root <none> and password file <none>
I20260928 06:33:18.329744 17033 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:18.329819 17033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:18.329964 17033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:18.331429 17033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/minicluster-data/master-0/data/instance:
uuid: "233bdeb8793943be9f32f7a0deb0953e"
format_stamp: "Formatted at 2026-09-28 06:33:18 on dist-test-slave-rrj5"
server_key: "732a01623c463b469e5291e61baf1eec"
server_key_iv: "091136a0d6fd7156dc39bc633b5999e8"
server_key_version: "encryptionkey@0"
I20260928 06:33:18.331830 17033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/minicluster-data/master-0/wal/instance:
uuid: "233bdeb8793943be9f32f7a0deb0953e"
format_stamp: "Formatted at 2026-09-28 06:33:18 on dist-test-slave-rrj5"
server_key: "732a01623c463b469e5291e61baf1eec"
server_key_iv: "091136a0d6fd7156dc39bc633b5999e8"
server_key_version: "encryptionkey@0"
I20260928 06:33:18.334345 17033 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260928 06:33:18.335276 17048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:18.335460 17033 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:18.335546 17033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/minicluster-data/master-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/minicluster-data/master-0/wal
uuid: "233bdeb8793943be9f32f7a0deb0953e"
format_stamp: "Formatted at 2026-09-28 06:33:18 on dist-test-slave-rrj5"
server_key: "732a01623c463b469e5291e61baf1eec"
server_key_iv: "091136a0d6fd7156dc39bc633b5999e8"
server_key_version: "encryptionkey@0"
I20260928 06:33:18.335675 17033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:18.347651 17033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:18.349459 17033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:18.349630 17033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:18.353979 17033 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.126:42551
I20260928 06:33:18.354029 17100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.126:42551 every 8 connection(s)
I20260928 06:33:18.354370 17033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
I20260928 06:33:18.355317 17101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:18.357717 17101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 233bdeb8793943be9f32f7a0deb0953e: Bootstrap starting.
I20260928 06:33:18.358300 17101 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 233bdeb8793943be9f32f7a0deb0953e: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:18.358598 17101 log.cc:824] T 00000000000000000000000000000000 P 233bdeb8793943be9f32f7a0deb0953e: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:18.359325 17101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 233bdeb8793943be9f32f7a0deb0953e: No bootstrap required, opened a new log
I20260928 06:33:18.360867 17101 raft_consensus.cc:348] T 00000000000000000000000000000000 P 233bdeb8793943be9f32f7a0deb0953e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "233bdeb8793943be9f32f7a0deb0953e" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 42551 } }
I20260928 06:33:18.361001 17101 raft_consensus.cc:374] T 00000000000000000000000000000000 P 233bdeb8793943be9f32f7a0deb0953e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:18.361023 17101 raft_consensus.cc:729] T 00000000000000000000000000000000 P 233bdeb8793943be9f32f7a0deb0953e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 233bdeb8793943be9f32f7a0deb0953e, State: Initialized, Role: FOLLOWER
I20260928 06:33:18.361214 17101 consensus_queue.cc:260] T 00000000000000000000000000000000 P 233bdeb8793943be9f32f7a0deb0953e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "233bdeb8793943be9f32f7a0deb0953e" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 42551 } }
I20260928 06:33:18.361295 17101 raft_consensus.cc:388] T 00000000000000000000000000000000 P 233bdeb8793943be9f32f7a0deb0953e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:33:18.361346 17101 raft_consensus.cc:482] T 00000000000000000000000000000000 P 233bdeb8793943be9f32f7a0deb0953e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:33:18.361394 17101 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 233bdeb8793943be9f32f7a0deb0953e [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:18.362001 17101 raft_consensus.cc:504] T 00000000000000000000000000000000 P 233bdeb8793943be9f32f7a0deb0953e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "233bdeb8793943be9f32f7a0deb0953e" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 42551 } }
I20260928 06:33:18.362115 17101 leader_election.cc:302] T 00000000000000000000000000000000 P 233bdeb8793943be9f32f7a0deb0953e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 233bdeb8793943be9f32f7a0deb0953e; no voters:
I20260928 06:33:18.362279 17101 leader_election.cc:288] T 00000000000000000000000000000000 P 233bdeb8793943be9f32f7a0deb0953e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:33:18.362309 17104 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 233bdeb8793943be9f32f7a0deb0953e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:33:18.362470 17104 raft_consensus.cc:686] T 00000000000000000000000000000000 P 233bdeb8793943be9f32f7a0deb0953e [term 1 LEADER]: Becoming Leader. State: Replica: 233bdeb8793943be9f32f7a0deb0953e, State: Running, Role: LEADER
I20260928 06:33:18.362540 17101 sys_catalog.cc:564] T 00000000000000000000000000000000 P 233bdeb8793943be9f32f7a0deb0953e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:33:18.362602 17104 consensus_queue.cc:237] T 00000000000000000000000000000000 P 233bdeb8793943be9f32f7a0deb0953e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "233bdeb8793943be9f32f7a0deb0953e" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 42551 } }
I20260928 06:33:18.363093 17106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 233bdeb8793943be9f32f7a0deb0953e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 233bdeb8793943be9f32f7a0deb0953e. Latest consensus state: current_term: 1 leader_uuid: "233bdeb8793943be9f32f7a0deb0953e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "233bdeb8793943be9f32f7a0deb0953e" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 42551 } } }
I20260928 06:33:18.363181 17106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 233bdeb8793943be9f32f7a0deb0953e [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:18.363253 17105 sys_catalog.cc:455] T 00000000000000000000000000000000 P 233bdeb8793943be9f32f7a0deb0953e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "233bdeb8793943be9f32f7a0deb0953e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "233bdeb8793943be9f32f7a0deb0953e" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 42551 } } }
I20260928 06:33:18.363336 17105 sys_catalog.cc:458] T 00000000000000000000000000000000 P 233bdeb8793943be9f32f7a0deb0953e [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:18.363557 17113 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:33:18.363945 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 17033
I20260928 06:33:18.364053 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/minicluster-data/master-0/wal/instance
I20260928 06:33:18.364130 17113 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:33:18.364789 17013 external_mini_cluster.cc:1428] Setting key 59002b48166c116cb478bbcc318534c6
I20260928 06:33:18.366365 17113 catalog_manager.cc:1347] Generated new cluster ID: 990b574d47f2491faa477209395ddf4c
I20260928 06:33:18.366421 17113 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:33:18 dist-test-slave-rrj5 krb5kdc[17020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:33:18.387117 17113 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:33:18.387300 17123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:18.365716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44776 (local address 127.16.157.126:42551)
0928 06:33:18.365888 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:18.365898 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:18.368437 (+ 2539us) server_negotiation.cc:407] Connection header received
0928 06:33:18.370317 (+ 1880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:18.370325 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:18.370379 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:18.370478 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:18.372651 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:18.374438 (+ 1787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:18.376169 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:18.376833 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:18.376911 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:18.382069 (+ 5158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:18.382086 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:18.382086 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:18.382099 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:18.384701 (+ 2602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:18.385268 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:18.385271 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:18.385272 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:18.385390 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:18.385772 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:18.385773 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:18.385774 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:18.386163 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:18.386332 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:18.386839 (+ 507us) server_negotiation.cc:299] Negotiation successful
0928 06:33:18.386996 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":99,"threads_started":1}
I20260928 06:33:18.387888 17113 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:33:18.393308 17113 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 233bdeb8793943be9f32f7a0deb0953e: Generated new TSK 0
I20260928 06:33:18.393514 17113 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.157.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab.
Entry for principal kudu/127.16.157.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab.
I20260928 06:33:18.422958 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.65:0
--local_ip_for_outbound_sockets=127.16.157.65
--webserver_interface=127.16.157.65
--webserver_port=0
--tserver_master_addrs=127.16.157.126:42551
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab
--principal=kudu/127.16.157.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:45483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:18.525308 17127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:18.525523 17127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:18.525564 17127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:18.526714 17127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:18.526783 17127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:18.526829 17127 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.65
I20260928 06:33:18.528370 17127 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:45483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab
--principal=kudu/127.16.157.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.157.65
--webserver_port=0
--tserver_master_addrs=127.16.157.126:42551
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.65
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:18.528621 17127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:18.528841 17127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:18.533795 17135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:18.533834 17133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:18.533874 17132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:18.534344 17127 server_base.cc:1034] running on GCE node
I20260928 06:33:18.534560 17127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:18.534821 17127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:18.535996 17127 hybrid_clock.cc:648] HybridClock initialized: now 1790577198535960 us; error 34 us; skew 500 ppm
Sep 28 06:33:18 dist-test-slave-rrj5 krb5kdc[17020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577198, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:18.538596 17127 init.cc:377] Logged in from keytab as kudu/127.16.157.65@KRBTEST.COM (short username kudu)
I20260928 06:33:18.539075 17127 webserver.cc:460] Webserver started at http://127.16.157.65:38049/ using document root <none> and password file <none>
I20260928 06:33:18.539341 17127 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:18.539412 17127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:18.539546 17127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:18.540634 17127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/minicluster-data/ts-0/data/instance:
uuid: "8a6fe7dd934f417888b809dcf3f10b91"
format_stamp: "Formatted at 2026-09-28 06:33:18 on dist-test-slave-rrj5"
server_key: "de1a7b01367416419d0e98255684139f"
server_key_iv: "9457dc64f4b35efcd3a3d4bbfb73bff5"
server_key_version: "encryptionkey@0"
I20260928 06:33:18.541049 17127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/minicluster-data/ts-0/wal/instance:
uuid: "8a6fe7dd934f417888b809dcf3f10b91"
format_stamp: "Formatted at 2026-09-28 06:33:18 on dist-test-slave-rrj5"
server_key: "de1a7b01367416419d0e98255684139f"
server_key_iv: "9457dc64f4b35efcd3a3d4bbfb73bff5"
server_key_version: "encryptionkey@0"
I20260928 06:33:18.542558 17127 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:33:18.543349 17142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:18.543533 17127 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:33:18.543637 17127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/minicluster-data/ts-0/wal
uuid: "8a6fe7dd934f417888b809dcf3f10b91"
format_stamp: "Formatted at 2026-09-28 06:33:18 on dist-test-slave-rrj5"
server_key: "de1a7b01367416419d0e98255684139f"
server_key_iv: "9457dc64f4b35efcd3a3d4bbfb73bff5"
server_key_version: "encryptionkey@0"
I20260928 06:33:18.543723 17127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:18.557291 17127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:18.559654 17127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:18.559813 17127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:18.560035 17127 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:33:18.560379 17127 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:33:18.560432 17127 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:18.560473 17127 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:33:18.560500 17127 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:18.567324 17127 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.65:46869
I20260928 06:33:18.567389 17255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.65:46869 every 8 connection(s)
I20260928 06:33:18.568454 17127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
Sep 28 06:33:18 dist-test-slave-rrj5 krb5kdc[17020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577198, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.65@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:33:18.578223 17123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:18.569221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:37457 (local address 127.16.157.126:42551)
0928 06:33:18.569265 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:18.569267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:18.569451 (+ 184us) server_negotiation.cc:407] Connection header received
0928 06:33:18.570028 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:18.570029 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:18.570042 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:18.570106 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:18.571062 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:18.571568 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:18.572562 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:18.572776 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:18.572853 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:18.575310 (+ 2457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:18.575312 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:18.575312 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:18.575319 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:18.576746 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:18.577221 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:18.577221 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:18.577222 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:18.577273 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:18.577539 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:18.577540 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:18.577540 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:18.577697 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:18.577749 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:18.578093 (+ 344us) server_negotiation.cc:299] Negotiation successful
0928 06:33:18.578157 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:33:18.578460 17258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:18.569179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:42551 (local address 127.16.157.65:37457)
0928 06:33:18.569322 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:18.569368 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:18.569923 (+ 555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:18.570206 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:18.570210 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:18.570754 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:18.570960 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:18.570964 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:18.571665 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:18.571666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:18.572468 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:18.572470 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:18.572860 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:18.572861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:18.572989 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:18.573481 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:18.573489 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:18.575213 (+ 1724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:18.576830 (+ 1617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:18.576838 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:18.576840 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:18.577138 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:18.577339 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:18.577340 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:18.577340 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:18.577474 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:18.577757 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:18.577764 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:18.577937 (+ 173us) client_negotiation.cc:769] Sending connection context
0928 06:33:18.578073 (+ 136us) client_negotiation.cc:240] Negotiation successful
0928 06:33:18.578232 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":71,"threads_started":1}
I20260928 06:33:18.578482 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 17127
I20260928 06:33:18.578560 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/minicluster-data/ts-0/wal/instance
I20260928 06:33:18.578719 17013 external_mini_cluster.cc:1428] Setting key f430512b1c5e3c6bb724b20f7cae39b5
I20260928 06:33:18.579144 17256 heartbeater.cc:347] Connected to a master server at 127.16.157.126:42551
I20260928 06:33:18.579243 17256 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:18.579496 17256 heartbeater.cc:511] Master 127.16.157.126:42551 requested a full tablet report, sending...
I20260928 06:33:18.580108 17065 ts_manager.cc:194] Registered new tserver with Master: 8a6fe7dd934f417888b809dcf3f10b91 (127.16.157.65:46869)
I20260928 06:33:18.581061 17065 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.157.65@KRBTEST.COM'} at 127.16.157.65:37457
WARNING: no policy specified for kudu/127.16.157.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab.
Entry for principal kudu/127.16.157.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab.
I20260928 06:33:18.604492 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.66:0
--local_ip_for_outbound_sockets=127.16.157.66
--webserver_interface=127.16.157.66
--webserver_port=0
--tserver_master_addrs=127.16.157.126:42551
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab
--principal=kudu/127.16.157.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:45483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:18.688602 17261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:18.688856 17261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:18.688882 17261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:18.690060 17261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:18.690099 17261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:18.690132 17261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.66
I20260928 06:33:18.691532 17261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:45483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab
--principal=kudu/127.16.157.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.157.66
--webserver_port=0
--tserver_master_addrs=127.16.157.126:42551
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.66
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:18.691773 17261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:18.691980 17261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:18.697095 17267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:18.697132 17261 server_base.cc:1034] running on GCE node
W20260928 06:33:18.697119 17266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:18.697100 17269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:18.697721 17261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:18.698173 17261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:18.699375 17261 hybrid_clock.cc:648] HybridClock initialized: now 1790577198699368 us; error 39 us; skew 500 ppm
Sep 28 06:33:18 dist-test-slave-rrj5 krb5kdc[17020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577198, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:18.702684 17261 init.cc:377] Logged in from keytab as kudu/127.16.157.66@KRBTEST.COM (short username kudu)
I20260928 06:33:18.703147 17261 webserver.cc:460] Webserver started at http://127.16.157.66:33887/ using document root <none> and password file <none>
I20260928 06:33:18.703393 17261 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:18.703459 17261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:18.703615 17261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:18.704650 17261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/minicluster-data/ts-1/data/instance:
uuid: "3cfba05b23d440e2b21fe8507cfd9781"
format_stamp: "Formatted at 2026-09-28 06:33:18 on dist-test-slave-rrj5"
server_key: "b67b29539893a3385defc47573583753"
server_key_iv: "43b1487624b698454ad103cbde4e940b"
server_key_version: "encryptionkey@0"
I20260928 06:33:18.705006 17261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/minicluster-data/ts-1/wal/instance:
uuid: "3cfba05b23d440e2b21fe8507cfd9781"
format_stamp: "Formatted at 2026-09-28 06:33:18 on dist-test-slave-rrj5"
server_key: "b67b29539893a3385defc47573583753"
server_key_iv: "43b1487624b698454ad103cbde4e940b"
server_key_version: "encryptionkey@0"
I20260928 06:33:18.706466 17261 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:33:18.707356 17276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:18.707541 17261 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:18.707662 17261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/minicluster-data/ts-1/wal
uuid: "3cfba05b23d440e2b21fe8507cfd9781"
format_stamp: "Formatted at 2026-09-28 06:33:18 on dist-test-slave-rrj5"
server_key: "b67b29539893a3385defc47573583753"
server_key_iv: "43b1487624b698454ad103cbde4e940b"
server_key_version: "encryptionkey@0"
I20260928 06:33:18.707759 17261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:18.722560 17261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:18.724779 17261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:18.724946 17261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:18.725198 17261 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:33:18.725484 17261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:33:18.725536 17261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:18.725596 17261 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:33:18.725647 17261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:18.731858 17261 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.66:43219
I20260928 06:33:18.731904 17389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.66:43219 every 8 connection(s)
I20260928 06:33:18.732192 17261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
Sep 28 06:33:18 dist-test-slave-rrj5 krb5kdc[17020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577198, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.66@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:33:18.739467 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 17261
I20260928 06:33:18.739566 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/minicluster-data/ts-1/wal/instance
I20260928 06:33:18.739743 17013 external_mini_cluster.cc:1428] Setting key 9c510379b2b9891277c5ee5f59721d79
I20260928 06:33:18.743040 17123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:18.732947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.66:47499 (local address 127.16.157.126:42551)
0928 06:33:18.733021 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:18.733023 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:18.733260 (+ 237us) server_negotiation.cc:407] Connection header received
0928 06:33:18.733922 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:18.733924 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:18.733934 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:18.733982 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:18.735188 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:18.735689 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:18.737130 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:18.737339 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:18.737399 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:18.739766 (+ 2367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:18.739768 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:18.739769 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:18.739777 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:18.741376 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:18.741927 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:18.741928 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:18.741929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:18.742001 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:18.742308 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:18.742309 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:18.742309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:18.742457 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:18.742526 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:18.742873 (+ 347us) server_negotiation.cc:299] Negotiation successful
0928 06:33:18.742954 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 06:33:18.743444 17392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:18.732917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:42551 (local address 127.16.157.66:47499)
0928 06:33:18.733115 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:18.733162 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:18.733811 (+ 649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:18.734060 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:18.734064 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:18.734785 (+ 721us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:18.735053 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:18.735058 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:18.735791 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:18.735792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:18.737043 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:18.737046 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:18.737428 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:18.737429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:18.737605 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:18.738089 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:18.738097 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:18.739626 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:18.741482 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:18.741490 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:18.741492 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:18.741819 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:18.742080 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:18.742080 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:18.742081 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:18.742215 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:18.742533 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:18.742540 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:18.742707 (+ 167us) client_negotiation.cc:769] Sending connection context
0928 06:33:18.742818 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 06:33:18.742999 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":107,"threads_started":1}
I20260928 06:33:18.746094 17390 heartbeater.cc:347] Connected to a master server at 127.16.157.126:42551
I20260928 06:33:18.746197 17390 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:18.746382 17390 heartbeater.cc:511] Master 127.16.157.126:42551 requested a full tablet report, sending...
I20260928 06:33:18.746769 17065 ts_manager.cc:194] Registered new tserver with Master: 3cfba05b23d440e2b21fe8507cfd9781 (127.16.157.66:43219)
I20260928 06:33:18.747301 17065 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.157.66@KRBTEST.COM'} at 127.16.157.66:47499
WARNING: no policy specified for kudu/127.16.157.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab.
Entry for principal kudu/127.16.157.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab.
I20260928 06:33:18.766005 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.67:0
--local_ip_for_outbound_sockets=127.16.157.67
--webserver_interface=127.16.157.67
--webserver_port=0
--tserver_master_addrs=127.16.157.126:42551
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab
--principal=kudu/127.16.157.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:45483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:18.847668 17395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:18.847934 17395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:18.847973 17395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:18.849161 17395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:18.849229 17395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:18.849275 17395 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.67
I20260928 06:33:18.850768 17395 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:45483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab
--principal=kudu/127.16.157.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.157.67
--webserver_port=0
--tserver_master_addrs=127.16.157.126:42551
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17395
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.67
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:18.851053 17395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:18.851284 17395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:18.858179 17400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:18.858201 17403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:18.858201 17401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:18.858646 17395 server_base.cc:1034] running on GCE node
I20260928 06:33:18.858850 17395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:18.859083 17395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:18.860260 17395 hybrid_clock.cc:648] HybridClock initialized: now 1790577198860243 us; error 36 us; skew 500 ppm
Sep 28 06:33:18 dist-test-slave-rrj5 krb5kdc[17020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577198, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:18.863351 17395 init.cc:377] Logged in from keytab as kudu/127.16.157.67@KRBTEST.COM (short username kudu)
I20260928 06:33:18.864046 17395 webserver.cc:460] Webserver started at http://127.16.157.67:41797/ using document root <none> and password file <none>
I20260928 06:33:18.864283 17395 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:18.864348 17395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:18.864522 17395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:18.865649 17395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/minicluster-data/ts-2/data/instance:
uuid: "065200e9fe794145882b45c0a187dcb8"
format_stamp: "Formatted at 2026-09-28 06:33:18 on dist-test-slave-rrj5"
server_key: "9076896e316ad1fe68f05c934b6a9002"
server_key_iv: "255a287ea936c0fe36b2e662d0d8d5d9"
server_key_version: "encryptionkey@0"
I20260928 06:33:18.866021 17395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/minicluster-data/ts-2/wal/instance:
uuid: "065200e9fe794145882b45c0a187dcb8"
format_stamp: "Formatted at 2026-09-28 06:33:18 on dist-test-slave-rrj5"
server_key: "9076896e316ad1fe68f05c934b6a9002"
server_key_iv: "255a287ea936c0fe36b2e662d0d8d5d9"
server_key_version: "encryptionkey@0"
I20260928 06:33:18.867403 17395 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 06:33:18.868423 17410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:18.868630 17395 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:33:18.868733 17395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/minicluster-data/ts-2/wal
uuid: "065200e9fe794145882b45c0a187dcb8"
format_stamp: "Formatted at 2026-09-28 06:33:18 on dist-test-slave-rrj5"
server_key: "9076896e316ad1fe68f05c934b6a9002"
server_key_iv: "255a287ea936c0fe36b2e662d0d8d5d9"
server_key_version: "encryptionkey@0"
I20260928 06:33:18.868810 17395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:18.881358 17395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:18.883977 17395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:18.884167 17395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:18.884430 17395 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:33:18.884776 17395 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:33:18.884836 17395 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:18.884886 17395 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:33:18.884920 17395 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:18.891954 17395 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.67:37771
I20260928 06:33:18.891999 17523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.67:37771 every 8 connection(s)
I20260928 06:33:18.892377 17395 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
Sep 28 06:33:18 dist-test-slave-rrj5 krb5kdc[17020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577198, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.67@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:33:18.901199 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 17395
I20260928 06:33:18.901320 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577196946045-17013-0/minicluster-data/ts-2/wal/instance
I20260928 06:33:18.901501 17013 external_mini_cluster.cc:1428] Setting key ba5ca3441b40fbd442da76b96140ba28
I20260928 06:33:18.902509 17123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:18.893185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:51337 (local address 127.16.157.126:42551)
0928 06:33:18.893229 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:18.893231 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:18.893470 (+ 239us) server_negotiation.cc:407] Connection header received
0928 06:33:18.894015 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:18.894016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:18.894027 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:18.894116 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:18.895104 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:18.895613 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:18.896634 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:18.896835 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:18.896908 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:18.899324 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:18.899326 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:18.899326 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:18.899333 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:18.900771 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:18.901313 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:18.901314 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:18.901315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:18.901379 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:18.901709 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:18.901710 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:18.901711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:18.901881 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:18.901969 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:18.902335 (+ 366us) server_negotiation.cc:299] Negotiation successful
0928 06:33:18.902422 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:33:18.902729 17526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:18.893168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:42551 (local address 127.16.157.67:51337)
0928 06:33:18.893333 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:18.893374 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:18.893922 (+ 548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:18.894200 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:18.894205 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:18.894800 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:18.894995 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:18.895000 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:18.895717 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:18.895718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:18.896529 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:18.896531 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:18.896929 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:18.896930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:18.897068 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:18.897652 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:18.897661 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:18.899219 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:18.900861 (+ 1642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:18.900869 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:18.900871 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:18.901214 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:18.901455 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:18.901456 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:18.901456 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:18.901624 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:18.901961 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:18.901967 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:18.902151 (+ 184us) client_negotiation.cc:769] Sending connection context
0928 06:33:18.902289 (+ 138us) client_negotiation.cc:240] Negotiation successful
0928 06:33:18.902461 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":91,"threads_started":1}
I20260928 06:33:18.903283 17524 heartbeater.cc:347] Connected to a master server at 127.16.157.126:42551
I20260928 06:33:18.903383 17524 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:18.903565 17524 heartbeater.cc:511] Master 127.16.157.126:42551 requested a full tablet report, sending...
I20260928 06:33:18.903929 17065 ts_manager.cc:194] Registered new tserver with Master: 065200e9fe794145882b45c0a187dcb8 (127.16.157.67:37771)
I20260928 06:33:18.904350 17065 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.157.67@KRBTEST.COM'} at 127.16.157.67:51337
I20260928 06:33:18.913846 17013 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 06:33:18 dist-test-slave-rrj5 krb5kdc[17020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577198, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 06:33:18 dist-test-slave-rrj5 krb5kdc[17020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577198, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.157.65@KRBTEST.COM
I20260928 06:33:18.943862 17531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:18.933554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36726 (local address 127.16.157.65:46869)
0928 06:33:18.933767 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:18.933772 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:18.933784 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:33:18.933812 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:18.933815 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:18.933830 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:18.933910 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:18.934717 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:18.935453 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:18.936648 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:18.937220 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:18.937311 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:18.940285 (+ 2974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:18.940295 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:18.940296 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:18.940306 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:18.942577 (+ 2271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:18.942940 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:18.942944 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:18.942945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:18.943019 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:18.943211 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:18.943213 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:18.943213 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:18.943405 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:18.943583 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:18.943631 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 06:33:18.943719 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":115,"threads_started":1}
W20260928 06:33:18.944315 17170 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36726
Sep 28 06:33:18 dist-test-slave-rrj5 krb5kdc[17020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 06:33:18 dist-test-slave-rrj5 krb5kdc[17020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.157.65@KRBTEST.COM
I20260928 06:33:18.968854 17531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:18.960546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36742 (local address 127.16.157.65:46869)
0928 06:33:18.960602 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:18.960604 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:18.960694 (+ 90us) server_negotiation.cc:407] Connection header received
0928 06:33:18.960775 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:18.960776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:18.960787 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:18.960851 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:18.961455 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:18.962029 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:18.963174 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:18.963475 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:18.963534 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:18.966258 (+ 2724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:18.966260 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:18.966260 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:18.966268 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:18.967675 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:18.968069 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:18.968070 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:18.968070 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:18.968129 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:18.968346 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:18.968347 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:18.968347 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:18.968448 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:18.968575 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:18.968669 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 06:33:18.968747 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 06:33:18.969422 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 17127
I20260928 06:33:18.974635 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 17261
I20260928 06:33:18.979272 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 17395
I20260928 06:33:18.984942 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 17033
2026-09-28T06:33:18Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2047 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:33:19 dist-test-slave-rrj5 krb5kdc[17538](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:33:19 dist-test-slave-rrj5 krb5kdc[17538](info): set up 2 sockets
Sep 28 06:33:19 dist-test-slave-rrj5 krb5kdc[17538](info): commencing operation
krb5kdc: starting...
W20260928 06:33:20.049928 17013 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Sep 28 06:33:20 dist-test-slave-rrj5 krb5kdc[17538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:33:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:33:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.157.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab.
Entry for principal kudu/127.16.157.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab.
I20260928 06:33:20.169005 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:40475
--webserver_interface=127.16.157.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=kudu/127.16.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:42331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.157.126:40475
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:20.250532 17551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:20.250766 17551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:20.250804 17551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:20.251972 17551 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:33:20.252038 17551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:20.252063 17551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:20.252090 17551 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:33:20.252122 17551 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:33:20.253634 17551 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:42331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.157.126:40475
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=kudu/127.16.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:40475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:20.253911 17551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:20.254146 17551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:20.258826 17559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:20.258875 17557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:20.259117 17551 server_base.cc:1034] running on GCE node
W20260928 06:33:20.259136 17556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:20.259490 17551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:20.259802 17551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:20.260964 17551 hybrid_clock.cc:648] HybridClock initialized: now 1790577200260952 us; error 27 us; skew 500 ppm
Sep 28 06:33:20 dist-test-slave-rrj5 krb5kdc[17538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577200, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:20.263777 17551 init.cc:377] Logged in from keytab as kudu/127.16.157.126@KRBTEST.COM (short username kudu)
I20260928 06:33:20.264254 17551 webserver.cc:460] Webserver started at http://127.16.157.126:44827/ using document root <none> and password file <none>
I20260928 06:33:20.264484 17551 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:20.264554 17551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:20.264703 17551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:20.265796 17551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/minicluster-data/master-0/data/instance:
uuid: "0d8aca5f238c4b30848612fdb4b67786"
format_stamp: "Formatted at 2026-09-28 06:33:20 on dist-test-slave-rrj5"
server_key: "300bad4a46c5a32263d9445adfbbc9b0"
server_key_iv: "cca6cab83f7f5e1c876e4878f51b3f33"
server_key_version: "encryptionkey@0"
I20260928 06:33:20.266172 17551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/minicluster-data/master-0/wal/instance:
uuid: "0d8aca5f238c4b30848612fdb4b67786"
format_stamp: "Formatted at 2026-09-28 06:33:20 on dist-test-slave-rrj5"
server_key: "300bad4a46c5a32263d9445adfbbc9b0"
server_key_iv: "cca6cab83f7f5e1c876e4878f51b3f33"
server_key_version: "encryptionkey@0"
I20260928 06:33:20.267514 17551 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:20.268271 17566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:20.268446 17551 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:20.268553 17551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/minicluster-data/master-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/minicluster-data/master-0/wal
uuid: "0d8aca5f238c4b30848612fdb4b67786"
format_stamp: "Formatted at 2026-09-28 06:33:20 on dist-test-slave-rrj5"
server_key: "300bad4a46c5a32263d9445adfbbc9b0"
server_key_iv: "cca6cab83f7f5e1c876e4878f51b3f33"
server_key_version: "encryptionkey@0"
I20260928 06:33:20.268626 17551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:20.284682 17551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:20.286755 17551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:20.286916 17551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:20.291222 17551 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.126:40475
I20260928 06:33:20.291260 17618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.126:40475 every 8 connection(s)
I20260928 06:33:20.291584 17551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
I20260928 06:33:20.292433 17619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:20.294301 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 17551
I20260928 06:33:20.294392 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/minicluster-data/master-0/wal/instance
I20260928 06:33:20.294562 17013 external_mini_cluster.cc:1428] Setting key 1a2187606cef890849f36e70f591e39a
I20260928 06:33:20.294871 17619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d8aca5f238c4b30848612fdb4b67786: Bootstrap starting.
I20260928 06:33:20.295584 17619 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d8aca5f238c4b30848612fdb4b67786: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:20.295907 17619 log.cc:824] T 00000000000000000000000000000000 P 0d8aca5f238c4b30848612fdb4b67786: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:20.296702 17619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d8aca5f238c4b30848612fdb4b67786: No bootstrap required, opened a new log
I20260928 06:33:20.299223 17619 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d8aca5f238c4b30848612fdb4b67786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8aca5f238c4b30848612fdb4b67786" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40475 } }
I20260928 06:33:20.299402 17619 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d8aca5f238c4b30848612fdb4b67786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:20.299438 17619 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d8aca5f238c4b30848612fdb4b67786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d8aca5f238c4b30848612fdb4b67786, State: Initialized, Role: FOLLOWER
I20260928 06:33:20.299594 17619 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d8aca5f238c4b30848612fdb4b67786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8aca5f238c4b30848612fdb4b67786" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40475 } }
I20260928 06:33:20.299677 17619 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d8aca5f238c4b30848612fdb4b67786 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:33:20.299710 17619 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d8aca5f238c4b30848612fdb4b67786 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:33:20.299757 17619 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d8aca5f238c4b30848612fdb4b67786 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:20.300408 17619 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d8aca5f238c4b30848612fdb4b67786 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8aca5f238c4b30848612fdb4b67786" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40475 } }
I20260928 06:33:20.300531 17619 leader_election.cc:302] T 00000000000000000000000000000000 P 0d8aca5f238c4b30848612fdb4b67786 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d8aca5f238c4b30848612fdb4b67786; no voters:
I20260928 06:33:20.300700 17619 leader_election.cc:288] T 00000000000000000000000000000000 P 0d8aca5f238c4b30848612fdb4b67786 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:33:20.300756 17624 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d8aca5f238c4b30848612fdb4b67786 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:33:20.300904 17624 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d8aca5f238c4b30848612fdb4b67786 [term 1 LEADER]: Becoming Leader. State: Replica: 0d8aca5f238c4b30848612fdb4b67786, State: Running, Role: LEADER
I20260928 06:33:20.300916 17619 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d8aca5f238c4b30848612fdb4b67786 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:33:20.300981 17624 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d8aca5f238c4b30848612fdb4b67786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8aca5f238c4b30848612fdb4b67786" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40475 } }
Sep 28 06:33:20 dist-test-slave-rrj5 krb5kdc[17538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:33:20.301299 17624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d8aca5f238c4b30848612fdb4b67786 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d8aca5f238c4b30848612fdb4b67786" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8aca5f238c4b30848612fdb4b67786" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40475 } } }
I20260928 06:33:20.301371 17624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d8aca5f238c4b30848612fdb4b67786 [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:20.301494 17624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d8aca5f238c4b30848612fdb4b67786 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d8aca5f238c4b30848612fdb4b67786. Latest consensus state: current_term: 1 leader_uuid: "0d8aca5f238c4b30848612fdb4b67786" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8aca5f238c4b30848612fdb4b67786" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40475 } } }
I20260928 06:33:20.301544 17624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d8aca5f238c4b30848612fdb4b67786 [sys.catalog]: This master's current role is: LEADER
W20260928 06:33:20.302884 17639 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0d8aca5f238c4b30848612fdb4b67786: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:33:20.303946 17639 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:33:20.304054 17630 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:33:20.304263 17630 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:33:20.305814 17622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:20.295093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34716 (local address 127.16.157.126:40475)
0928 06:33:20.295285 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:20.295296 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:20.295377 (+ 81us) server_negotiation.cc:407] Connection header received
0928 06:33:20.296014 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:20.296023 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:20.296090 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:20.296258 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:20.296900 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:20.297823 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:20.298577 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:20.298979 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:20.299097 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:20.301760 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:20.301764 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:20.301764 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:20.301776 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:20.303808 (+ 2032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:20.304251 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:20.304255 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:20.304255 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:20.304403 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:20.304654 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:20.304656 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:20.304656 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:20.304951 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:20.305095 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:20.305452 (+ 357us) server_negotiation.cc:299] Negotiation successful
0928 06:33:20.305618 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":84,"threads_started":1}
I20260928 06:33:20.310633 17630 catalog_manager.cc:1347] Generated new cluster ID: a2f19bc4c9d84bb2a9e0e49767c8bb61
I20260928 06:33:20.310689 17630 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:33:20.342936 17630 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:33:20.343660 17630 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:33:20.347421 17630 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d8aca5f238c4b30848612fdb4b67786: Generated new TSK 0
I20260928 06:33:20.347626 17630 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.157.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab.
Entry for principal kudu/127.16.157.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab.
I20260928 06:33:20.388051 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.65:0
--local_ip_for_outbound_sockets=127.16.157.65
--webserver_interface=127.16.157.65
--webserver_port=0
--tserver_master_addrs=127.16.157.126:40475
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab
--principal=kudu/127.16.157.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:42331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:20.492331 17645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:20.492514 17645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:20.492537 17645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:20.493726 17645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:20.493764 17645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:20.493790 17645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.65
I20260928 06:33:20.495237 17645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:42331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab
--principal=kudu/127.16.157.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.157.65
--webserver_port=0
--tserver_master_addrs=127.16.157.126:40475
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.65
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:20.495499 17645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:20.495710 17645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:20.500577 17651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:20.500752 17650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:20.500869 17645 server_base.cc:1034] running on GCE node
W20260928 06:33:20.500658 17653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:20.501134 17645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:20.501432 17645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:20.502600 17645 hybrid_clock.cc:648] HybridClock initialized: now 1790577200502585 us; error 39 us; skew 500 ppm
Sep 28 06:33:20 dist-test-slave-rrj5 krb5kdc[17538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577200, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:20.505090 17645 init.cc:377] Logged in from keytab as kudu/127.16.157.65@KRBTEST.COM (short username kudu)
I20260928 06:33:20.505560 17645 webserver.cc:460] Webserver started at http://127.16.157.65:34371/ using document root <none> and password file <none>
I20260928 06:33:20.505821 17645 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:20.505889 17645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:20.506017 17645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:20.507030 17645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/minicluster-data/ts-0/data/instance:
uuid: "b447e39c7f794753b08905085d1d4fa2"
format_stamp: "Formatted at 2026-09-28 06:33:20 on dist-test-slave-rrj5"
server_key: "019a417d3015f4952f5f7c3678cc332e"
server_key_iv: "8d4779c22900e428f45ee193878906cd"
server_key_version: "encryptionkey@0"
I20260928 06:33:20.507393 17645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/minicluster-data/ts-0/wal/instance:
uuid: "b447e39c7f794753b08905085d1d4fa2"
format_stamp: "Formatted at 2026-09-28 06:33:20 on dist-test-slave-rrj5"
server_key: "019a417d3015f4952f5f7c3678cc332e"
server_key_iv: "8d4779c22900e428f45ee193878906cd"
server_key_version: "encryptionkey@0"
I20260928 06:33:20.508740 17645 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:33:20.509511 17660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:20.509711 17645 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:33:20.509817 17645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/minicluster-data/ts-0/wal
uuid: "b447e39c7f794753b08905085d1d4fa2"
format_stamp: "Formatted at 2026-09-28 06:33:20 on dist-test-slave-rrj5"
server_key: "019a417d3015f4952f5f7c3678cc332e"
server_key_iv: "8d4779c22900e428f45ee193878906cd"
server_key_version: "encryptionkey@0"
I20260928 06:33:20.509902 17645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:20.529543 17645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:20.531680 17645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:20.531842 17645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:20.532095 17645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:33:20.532393 17645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:33:20.532445 17645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:20.532480 17645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:33:20.532506 17645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:20.538753 17645 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.65:40941
I20260928 06:33:20.538805 17773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.65:40941 every 8 connection(s)
I20260928 06:33:20.539106 17645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
I20260928 06:33:20.543902 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 17645
I20260928 06:33:20.544010 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/minicluster-data/ts-0/wal/instance
I20260928 06:33:20.544166 17013 external_mini_cluster.cc:1428] Setting key 2bb06b571a3fdebf0575561c52e61904
Sep 28 06:33:20 dist-test-slave-rrj5 krb5kdc[17538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577200, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.65@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:33:20.549813 17622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:20.539897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:48697 (local address 127.16.157.126:40475)
0928 06:33:20.539965 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:20.539967 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:20.540135 (+ 168us) server_negotiation.cc:407] Connection header received
0928 06:33:20.541044 (+ 909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:20.541046 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:20.541061 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:20.541162 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:20.541818 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:20.542688 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:20.543698 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:20.544017 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:20.544139 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:20.546988 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:20.546990 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:20.546990 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:20.546998 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:20.548397 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:20.548815 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:20.548816 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:20.548816 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:20.548881 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:20.549132 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:20.549133 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:20.549133 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:20.549256 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:20.549333 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:20.549637 (+ 304us) server_negotiation.cc:299] Negotiation successful
0928 06:33:20.549725 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 06:33:20.550010 17776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:20.539819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40475 (local address 127.16.157.65:48697)
0928 06:33:20.539998 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:20.540039 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:20.540930 (+ 891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:20.541214 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:20.541217 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:20.541454 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:20.541709 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:20.541714 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:20.542789 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:20.542790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:20.543586 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:20.543587 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:20.544254 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:20.544257 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:20.544453 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:20.544994 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:20.545007 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:20.546884 (+ 1877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:20.548497 (+ 1613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:20.548506 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:20.548508 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:20.548738 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:20.548948 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:20.548948 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:20.548949 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:20.549064 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:20.549324 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:20.549331 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:20.549471 (+ 140us) client_negotiation.cc:769] Sending connection context
0928 06:33:20.549575 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 06:33:20.549763 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":89,"threads_started":1}
I20260928 06:33:20.550581 17774 heartbeater.cc:347] Connected to a master server at 127.16.157.126:40475
I20260928 06:33:20.550701 17774 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:20.550897 17774 heartbeater.cc:511] Master 127.16.157.126:40475 requested a full tablet report, sending...
I20260928 06:33:20.551419 17583 ts_manager.cc:194] Registered new tserver with Master: b447e39c7f794753b08905085d1d4fa2 (127.16.157.65:40941)
I20260928 06:33:20.552026 17583 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.157.65@KRBTEST.COM'} at 127.16.157.65:48697
WARNING: no policy specified for kudu/127.16.157.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab.
Entry for principal kudu/127.16.157.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab.
I20260928 06:33:20.572152 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.66:0
--local_ip_for_outbound_sockets=127.16.157.66
--webserver_interface=127.16.157.66
--webserver_port=0
--tserver_master_addrs=127.16.157.126:40475
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab
--principal=kudu/127.16.157.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:42331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:20.663105 17779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:20.663296 17779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:20.663321 17779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:20.664436 17779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:20.664476 17779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:20.664503 17779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.66
I20260928 06:33:20.666018 17779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:42331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab
--principal=kudu/127.16.157.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.157.66
--webserver_port=0
--tserver_master_addrs=127.16.157.126:40475
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.66
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:20.666239 17779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:20.666466 17779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:20.671300 17784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:20.671288 17787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:20.671521 17779 server_base.cc:1034] running on GCE node
W20260928 06:33:20.671535 17785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:20.671837 17779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:20.672081 17779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:20.673261 17779 hybrid_clock.cc:648] HybridClock initialized: now 1790577200673248 us; error 28 us; skew 500 ppm
Sep 28 06:33:20 dist-test-slave-rrj5 krb5kdc[17538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577200, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:20.676051 17779 init.cc:377] Logged in from keytab as kudu/127.16.157.66@KRBTEST.COM (short username kudu)
I20260928 06:33:20.676540 17779 webserver.cc:460] Webserver started at http://127.16.157.66:37119/ using document root <none> and password file <none>
I20260928 06:33:20.676807 17779 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:20.676875 17779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:20.677011 17779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:20.678119 17779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/minicluster-data/ts-1/data/instance:
uuid: "b5715f9d837245fea4eddeac16b3de43"
format_stamp: "Formatted at 2026-09-28 06:33:20 on dist-test-slave-rrj5"
server_key: "98364627dea797b524e2ae3107b3f50b"
server_key_iv: "b8f18224e89ba3e515e08b49b40a9257"
server_key_version: "encryptionkey@0"
I20260928 06:33:20.678478 17779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/minicluster-data/ts-1/wal/instance:
uuid: "b5715f9d837245fea4eddeac16b3de43"
format_stamp: "Formatted at 2026-09-28 06:33:20 on dist-test-slave-rrj5"
server_key: "98364627dea797b524e2ae3107b3f50b"
server_key_iv: "b8f18224e89ba3e515e08b49b40a9257"
server_key_version: "encryptionkey@0"
I20260928 06:33:20.679800 17779 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:33:20.680544 17794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:20.680687 17779 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:33:20.680797 17779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/minicluster-data/ts-1/wal
uuid: "b5715f9d837245fea4eddeac16b3de43"
format_stamp: "Formatted at 2026-09-28 06:33:20 on dist-test-slave-rrj5"
server_key: "98364627dea797b524e2ae3107b3f50b"
server_key_iv: "b8f18224e89ba3e515e08b49b40a9257"
server_key_version: "encryptionkey@0"
I20260928 06:33:20.680871 17779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:20.700933 17779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:20.703485 17779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:20.703644 17779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:20.703908 17779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:33:20.704243 17779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:33:20.704315 17779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:20.704349 17779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:33:20.704375 17779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:20.711730 17779 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.66:37009
I20260928 06:33:20.712093 17779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
I20260928 06:33:20.712575 17907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.66:37009 every 8 connection(s)
I20260928 06:33:20.717314 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 17779
I20260928 06:33:20.717404 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/minicluster-data/ts-1/wal/instance
I20260928 06:33:20.717557 17013 external_mini_cluster.cc:1428] Setting key b21c6c0df48dbd9f0ec8841b2d99df21
Sep 28 06:33:20 dist-test-slave-rrj5 krb5kdc[17538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577200, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.66@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:33:20.722594 17622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:20.712987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.66:55593 (local address 127.16.157.126:40475)
0928 06:33:20.713031 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:20.713033 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:20.713285 (+ 252us) server_negotiation.cc:407] Connection header received
0928 06:33:20.714317 (+ 1032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:20.714319 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:20.714333 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:20.714410 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:20.714993 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:20.715792 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:20.716787 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:20.717028 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:20.717108 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:20.719484 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:20.719486 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:20.719487 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:20.719493 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:20.721182 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:20.721620 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:20.721621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:20.721622 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:20.721682 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:20.721950 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:20.721950 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:20.721951 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:20.722062 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:20.722133 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:20.722421 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 06:33:20.722506 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 06:33:20.722788 17910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:20.712961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40475 (local address 127.16.157.66:55593)
0928 06:33:20.713124 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:20.713162 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:20.714190 (+ 1028us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:20.714464 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:20.714467 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:20.714707 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:20.714893 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:20.714897 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:20.715891 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:20.715892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:20.716673 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:20.716675 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:20.717134 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:20.717135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:20.717280 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:20.717860 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:20.717869 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:20.719391 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:20.721263 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:20.721270 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:20.721273 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:20.721498 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:20.721764 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:20.721764 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:20.721765 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:20.721877 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:20.722125 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:20.722131 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:20.722289 (+ 158us) client_negotiation.cc:769] Sending connection context
0928 06:33:20.722397 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 06:33:20.722554 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":83,"threads_started":1}
I20260928 06:33:20.723285 17908 heartbeater.cc:347] Connected to a master server at 127.16.157.126:40475
I20260928 06:33:20.723392 17908 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:20.723577 17908 heartbeater.cc:511] Master 127.16.157.126:40475 requested a full tablet report, sending...
I20260928 06:33:20.723973 17583 ts_manager.cc:194] Registered new tserver with Master: b5715f9d837245fea4eddeac16b3de43 (127.16.157.66:37009)
I20260928 06:33:20.724367 17583 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.157.66@KRBTEST.COM'} at 127.16.157.66:55593
WARNING: no policy specified for kudu/127.16.157.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab.
Entry for principal kudu/127.16.157.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab.
I20260928 06:33:20.744184 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.67:0
--local_ip_for_outbound_sockets=127.16.157.67
--webserver_interface=127.16.157.67
--webserver_port=0
--tserver_master_addrs=127.16.157.126:40475
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab
--principal=kudu/127.16.157.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:42331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:20.851667 17913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:20.851871 17913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:20.851895 17913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:20.853046 17913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:20.853089 17913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:20.853121 17913 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.67
I20260928 06:33:20.854707 17913 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:42331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab
--principal=kudu/127.16.157.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.157.67
--webserver_port=0
--tserver_master_addrs=127.16.157.126:40475
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17913
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.67
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:20.854950 17913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:20.855209 17913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:20.860370 17919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:20.860342 17918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:20.860546 17913 server_base.cc:1034] running on GCE node
W20260928 06:33:20.860364 17921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:20.860893 17913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:20.861188 17913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:20.862349 17913 hybrid_clock.cc:648] HybridClock initialized: now 1790577200862327 us; error 38 us; skew 500 ppm
Sep 28 06:33:20 dist-test-slave-rrj5 krb5kdc[17538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577200, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:20.864820 17913 init.cc:377] Logged in from keytab as kudu/127.16.157.67@KRBTEST.COM (short username kudu)
I20260928 06:33:20.865417 17913 webserver.cc:460] Webserver started at http://127.16.157.67:45295/ using document root <none> and password file <none>
I20260928 06:33:20.865703 17913 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:20.865751 17913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:20.865903 17913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:20.866992 17913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/minicluster-data/ts-2/data/instance:
uuid: "1a6f2a62b2ba4b348ac425d72b7d46b1"
format_stamp: "Formatted at 2026-09-28 06:33:20 on dist-test-slave-rrj5"
server_key: "035ce5c1a91f1ea626adbcc1ed2db216"
server_key_iv: "f3c634e82ec7535af630b68ae6295416"
server_key_version: "encryptionkey@0"
I20260928 06:33:20.867369 17913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/minicluster-data/ts-2/wal/instance:
uuid: "1a6f2a62b2ba4b348ac425d72b7d46b1"
format_stamp: "Formatted at 2026-09-28 06:33:20 on dist-test-slave-rrj5"
server_key: "035ce5c1a91f1ea626adbcc1ed2db216"
server_key_iv: "f3c634e82ec7535af630b68ae6295416"
server_key_version: "encryptionkey@0"
I20260928 06:33:20.868927 17913 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:33:20.869730 17928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:20.869903 17913 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:33:20.869998 17913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/minicluster-data/ts-2/wal
uuid: "1a6f2a62b2ba4b348ac425d72b7d46b1"
format_stamp: "Formatted at 2026-09-28 06:33:20 on dist-test-slave-rrj5"
server_key: "035ce5c1a91f1ea626adbcc1ed2db216"
server_key_iv: "f3c634e82ec7535af630b68ae6295416"
server_key_version: "encryptionkey@0"
I20260928 06:33:20.870088 17913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:20.882637 17913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:20.885193 17913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:20.885381 17913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:20.885710 17913 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:33:20.886029 17913 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:33:20.886087 17913 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:20.886121 17913 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:33:20.886139 17913 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:20.892994 17913 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.67:39117
I20260928 06:33:20.893065 18041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.67:39117 every 8 connection(s)
I20260928 06:33:20.893394 17913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
I20260928 06:33:20.899428 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 17913
I20260928 06:33:20.899535 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577196946045-17013-0/minicluster-data/ts-2/wal/instance
I20260928 06:33:20.899700 17013 external_mini_cluster.cc:1428] Setting key 2976cfeb8335348c0c8796ebc707983c
Sep 28 06:33:20 dist-test-slave-rrj5 krb5kdc[17538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577200, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.67@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:33:20.903455 17622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:20.894199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:54185 (local address 127.16.157.126:40475)
0928 06:33:20.894242 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:20.894244 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:20.894449 (+ 205us) server_negotiation.cc:407] Connection header received
0928 06:33:20.895374 (+ 925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:20.895375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:20.895387 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:20.895459 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:20.896045 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:20.896736 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:20.897785 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:20.898040 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:20.898116 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:20.900443 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:20.900444 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:20.900444 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:20.900452 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:20.902044 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:20.902469 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:20.902470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:20.902470 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:20.902527 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:20.902820 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:20.902821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:20.902821 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:20.902970 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:20.903047 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:20.903288 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 06:33:20.903370 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 06:33:20.903555 18044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:20.894175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40475 (local address 127.16.157.67:54185)
0928 06:33:20.894337 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:20.894373 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:20.895262 (+ 889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:20.895517 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:20.895521 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:20.895756 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:20.895955 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:20.895959 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:20.896834 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:20.896835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:20.897706 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:20.897708 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:20.898127 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:20.898128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:20.898223 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:20.898739 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:20.898748 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:20.900358 (+ 1610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:20.902135 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:20.902142 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:20.902144 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:20.902376 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:20.902603 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:20.902604 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:20.902604 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:20.902736 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:20.903041 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:20.903046 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:20.903159 (+ 113us) client_negotiation.cc:769] Sending connection context
0928 06:33:20.903242 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 06:33:20.903364 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":88,"threads_started":1}
I20260928 06:33:20.904006 18042 heartbeater.cc:347] Connected to a master server at 127.16.157.126:40475
I20260928 06:33:20.904117 18042 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:20.904290 18042 heartbeater.cc:511] Master 127.16.157.126:40475 requested a full tablet report, sending...
I20260928 06:33:20.904747 17583 ts_manager.cc:194] Registered new tserver with Master: 1a6f2a62b2ba4b348ac425d72b7d46b1 (127.16.157.67:39117)
I20260928 06:33:20.905202 17583 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.157.67@KRBTEST.COM'} at 127.16.157.67:54185
I20260928 06:33:20.911435 17013 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:33:20.919829 17622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:20.913292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34720 (local address 127.16.157.126:40475)
0928 06:33:20.913356 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:20.913358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:20.913473 (+ 115us) server_negotiation.cc:407] Connection header received
0928 06:33:20.913527 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:20.913528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:20.913536 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:20.913639 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:20.914296 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:20.914763 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:20.915751 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:20.915938 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:20.916009 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:20.917384 (+ 1375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:20.917385 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:20.917385 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:20.917391 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:20.918729 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:20.919117 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:20.919118 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:20.919118 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:20.919166 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:20.919351 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:20.919351 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:20.919351 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:20.919484 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:20.919556 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:20.919702 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 06:33:20.919757 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 06:33:20.922405 17583 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34720:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:33:20.923964 17583 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:33:20.931115 18056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:20.926554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59576 (local address 127.16.157.67:39117)
0928 06:33:20.926967 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:20.926970 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:20.926980 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:33:20.927024 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:20.927025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:20.927037 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:20.927105 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:33:20.927336 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:20.928351 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:20.929870 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:20.930575 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:20.930626 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:20.930689 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:20.930993 (+ 304us) server_negotiation.cc:299] Negotiation successful
0928 06:33:20.931038 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":91,"threads_started":1}
I20260928 06:33:20.931339 18053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:20.926366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.67:39117 (local address 127.0.0.1:59576)
0928 06:33:20.926478 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:20.926515 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:20.926581 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:20.927140 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:20.927143 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:20.927146 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:33:20.927225 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:20.927227 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:20.928476 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:20.928476 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:20.929764 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:20.929767 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:20.930767 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:20.930768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:20.930861 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:20.930877 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 06:33:20.931206 (+ 329us) client_negotiation.cc:240] Negotiation successful
0928 06:33:20.931271 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":88,"threads_started":1}
I20260928 06:33:20.932237 18054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:20.926368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.65:40941 (local address 127.0.0.1:34204)
0928 06:33:20.927230 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:20.927241 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:20.927293 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:20.928016 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:20.928017 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:20.928019 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:33:20.928053 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:20.928054 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:20.929246 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:20.929247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:20.930588 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:20.930590 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:20.931990 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:20.931991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:20.932056 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:20.932059 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:33:20.932106 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 06:33:20.932146 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":840,"mutex_wait_us":13,"thread_start_us":65,"threads_started":1}
I20260928 06:33:20.932160 17976 tablet_service.cc:1467] Processing CreateTablet for tablet 09188ad29b434082950b5278e4ed0da6 (DEFAULT_TABLE table=test-table [id=35aa42c3edde4201969e62b794caf230]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:33:20.932452 18055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:20.926582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.66:37009 (local address 127.0.0.1:46256)
0928 06:33:20.927427 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:20.927437 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:20.927482 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:20.927690 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:20.927691 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:20.927693 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:33:20.927748 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:20.927749 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:20.929048 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:20.929049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:20.930587 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:20.930590 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:20.932123 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:20.932124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:20.932171 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:20.932174 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:33:20.932219 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 06:33:20.932249 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":826,"mutex_wait_us":61,"thread_start_us":69,"threads_started":1}
I20260928 06:33:20.932571 17976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09188ad29b434082950b5278e4ed0da6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:20.932698 18058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:20.926760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34204 (local address 127.16.157.65:40941)
0928 06:33:20.926920 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:20.926927 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:20.927789 (+ 862us) server_negotiation.cc:407] Connection header received
0928 06:33:20.927835 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:20.927837 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:20.927852 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:20.927908 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:33:20.928123 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:20.929141 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:20.930691 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:20.931827 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:20.931897 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:20.931951 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:20.932570 (+ 619us) server_negotiation.cc:299] Negotiation successful
0928 06:33:20.932625 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":91,"threads_started":1}
I20260928 06:33:20.932992 18057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:20.926731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46256 (local address 127.16.157.66:37009)
0928 06:33:20.926938 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:20.926943 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:20.927547 (+ 604us) server_negotiation.cc:407] Connection header received
0928 06:33:20.927598 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:20.927600 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:20.927612 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:20.927694 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:33:20.927839 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:20.928938 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:20.930691 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:20.931986 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:20.932044 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:20.932093 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:20.932476 (+ 383us) server_negotiation.cc:299] Negotiation successful
0928 06:33:20.932912 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":87,"threads_started":1}
I20260928 06:33:20.933357 17708 tablet_service.cc:1467] Processing CreateTablet for tablet 09188ad29b434082950b5278e4ed0da6 (DEFAULT_TABLE table=test-table [id=35aa42c3edde4201969e62b794caf230]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:33:20.933729 17708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09188ad29b434082950b5278e4ed0da6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:20.933868 17842 tablet_service.cc:1467] Processing CreateTablet for tablet 09188ad29b434082950b5278e4ed0da6 (DEFAULT_TABLE table=test-table [id=35aa42c3edde4201969e62b794caf230]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:33:20.934162 17842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09188ad29b434082950b5278e4ed0da6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:20.936190 18059 tablet_bootstrap.cc:492] T 09188ad29b434082950b5278e4ed0da6 P 1a6f2a62b2ba4b348ac425d72b7d46b1: Bootstrap starting.
I20260928 06:33:20.936753 18059 tablet_bootstrap.cc:654] T 09188ad29b434082950b5278e4ed0da6 P 1a6f2a62b2ba4b348ac425d72b7d46b1: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:20.937146 18060 tablet_bootstrap.cc:492] T 09188ad29b434082950b5278e4ed0da6 P b447e39c7f794753b08905085d1d4fa2: Bootstrap starting.
I20260928 06:33:20.937279 18059 log.cc:824] T 09188ad29b434082950b5278e4ed0da6 P 1a6f2a62b2ba4b348ac425d72b7d46b1: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:20.937718 18061 tablet_bootstrap.cc:492] T 09188ad29b434082950b5278e4ed0da6 P b5715f9d837245fea4eddeac16b3de43: Bootstrap starting.
I20260928 06:33:20.937839 18060 tablet_bootstrap.cc:654] T 09188ad29b434082950b5278e4ed0da6 P b447e39c7f794753b08905085d1d4fa2: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:20.938126 18059 tablet_bootstrap.cc:492] T 09188ad29b434082950b5278e4ed0da6 P 1a6f2a62b2ba4b348ac425d72b7d46b1: No bootstrap required, opened a new log
I20260928 06:33:20.938210 18059 ts_tablet_manager.cc:1397] T 09188ad29b434082950b5278e4ed0da6 P 1a6f2a62b2ba4b348ac425d72b7d46b1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:33:20.938359 18060 log.cc:824] T 09188ad29b434082950b5278e4ed0da6 P b447e39c7f794753b08905085d1d4fa2: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:20.938670 18061 tablet_bootstrap.cc:654] T 09188ad29b434082950b5278e4ed0da6 P b5715f9d837245fea4eddeac16b3de43: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:20.939225 18061 log.cc:824] T 09188ad29b434082950b5278e4ed0da6 P b5715f9d837245fea4eddeac16b3de43: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:20.939864 18060 tablet_bootstrap.cc:492] T 09188ad29b434082950b5278e4ed0da6 P b447e39c7f794753b08905085d1d4fa2: No bootstrap required, opened a new log
I20260928 06:33:20.939929 18060 ts_tablet_manager.cc:1397] T 09188ad29b434082950b5278e4ed0da6 P b447e39c7f794753b08905085d1d4fa2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 06:33:20.940163 18059 raft_consensus.cc:348] T 09188ad29b434082950b5278e4ed0da6 P 1a6f2a62b2ba4b348ac425d72b7d46b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b447e39c7f794753b08905085d1d4fa2" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 40941 } } peers { permanent_uuid: "1a6f2a62b2ba4b348ac425d72b7d46b1" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39117 } } peers { permanent_uuid: "b5715f9d837245fea4eddeac16b3de43" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 37009 } }
I20260928 06:33:20.940294 18059 raft_consensus.cc:374] T 09188ad29b434082950b5278e4ed0da6 P 1a6f2a62b2ba4b348ac425d72b7d46b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:20.940326 18059 raft_consensus.cc:729] T 09188ad29b434082950b5278e4ed0da6 P 1a6f2a62b2ba4b348ac425d72b7d46b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a6f2a62b2ba4b348ac425d72b7d46b1, State: Initialized, Role: FOLLOWER
I20260928 06:33:20.940450 18059 consensus_queue.cc:260] T 09188ad29b434082950b5278e4ed0da6 P 1a6f2a62b2ba4b348ac425d72b7d46b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b447e39c7f794753b08905085d1d4fa2" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 40941 } } peers { permanent_uuid: "1a6f2a62b2ba4b348ac425d72b7d46b1" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39117 } } peers { permanent_uuid: "b5715f9d837245fea4eddeac16b3de43" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 37009 } }
I20260928 06:33:20.940735 18042 heartbeater.cc:503] Master 127.16.157.126:40475 was elected leader, sending a full tablet report...
I20260928 06:33:20.941083 18059 ts_tablet_manager.cc:1428] T 09188ad29b434082950b5278e4ed0da6 P 1a6f2a62b2ba4b348ac425d72b7d46b1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:33:20.941816 18060 raft_consensus.cc:348] T 09188ad29b434082950b5278e4ed0da6 P b447e39c7f794753b08905085d1d4fa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b447e39c7f794753b08905085d1d4fa2" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 40941 } } peers { permanent_uuid: "1a6f2a62b2ba4b348ac425d72b7d46b1" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39117 } } peers { permanent_uuid: "b5715f9d837245fea4eddeac16b3de43" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 37009 } }
I20260928 06:33:20.941977 18060 raft_consensus.cc:374] T 09188ad29b434082950b5278e4ed0da6 P b447e39c7f794753b08905085d1d4fa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:20.942006 18060 raft_consensus.cc:729] T 09188ad29b434082950b5278e4ed0da6 P b447e39c7f794753b08905085d1d4fa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b447e39c7f794753b08905085d1d4fa2, State: Initialized, Role: FOLLOWER
I20260928 06:33:20.942111 18060 consensus_queue.cc:260] T 09188ad29b434082950b5278e4ed0da6 P b447e39c7f794753b08905085d1d4fa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b447e39c7f794753b08905085d1d4fa2" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 40941 } } peers { permanent_uuid: "1a6f2a62b2ba4b348ac425d72b7d46b1" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39117 } } peers { permanent_uuid: "b5715f9d837245fea4eddeac16b3de43" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 37009 } }
I20260928 06:33:20.942214 18061 tablet_bootstrap.cc:492] T 09188ad29b434082950b5278e4ed0da6 P b5715f9d837245fea4eddeac16b3de43: No bootstrap required, opened a new log
I20260928 06:33:20.942284 18061 ts_tablet_manager.cc:1397] T 09188ad29b434082950b5278e4ed0da6 P b5715f9d837245fea4eddeac16b3de43: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260928 06:33:20.942355 18060 ts_tablet_manager.cc:1428] T 09188ad29b434082950b5278e4ed0da6 P b447e39c7f794753b08905085d1d4fa2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:33:20.942409 17774 heartbeater.cc:503] Master 127.16.157.126:40475 was elected leader, sending a full tablet report...
I20260928 06:33:20.944002 18061 raft_consensus.cc:348] T 09188ad29b434082950b5278e4ed0da6 P b5715f9d837245fea4eddeac16b3de43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b447e39c7f794753b08905085d1d4fa2" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 40941 } } peers { permanent_uuid: "1a6f2a62b2ba4b348ac425d72b7d46b1" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39117 } } peers { permanent_uuid: "b5715f9d837245fea4eddeac16b3de43" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 37009 } }
I20260928 06:33:20.944128 18061 raft_consensus.cc:374] T 09188ad29b434082950b5278e4ed0da6 P b5715f9d837245fea4eddeac16b3de43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:20.944149 18061 raft_consensus.cc:729] T 09188ad29b434082950b5278e4ed0da6 P b5715f9d837245fea4eddeac16b3de43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5715f9d837245fea4eddeac16b3de43, State: Initialized, Role: FOLLOWER
I20260928 06:33:20.944229 18061 consensus_queue.cc:260] T 09188ad29b434082950b5278e4ed0da6 P b5715f9d837245fea4eddeac16b3de43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b447e39c7f794753b08905085d1d4fa2" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 40941 } } peers { permanent_uuid: "1a6f2a62b2ba4b348ac425d72b7d46b1" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39117 } } peers { permanent_uuid: "b5715f9d837245fea4eddeac16b3de43" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 37009 } }
I20260928 06:33:20.944422 18061 ts_tablet_manager.cc:1428] T 09188ad29b434082950b5278e4ed0da6 P b5715f9d837245fea4eddeac16b3de43: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 06:33:20.944505 17908 heartbeater.cc:503] Master 127.16.157.126:40475 was elected leader, sending a full tablet report...
W20260928 06:33:20.962704 17909 tablet.cc:2369] T 09188ad29b434082950b5278e4ed0da6 P b5715f9d837245fea4eddeac16b3de43: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:33:21.000950 18067 raft_consensus.cc:482] T 09188ad29b434082950b5278e4ed0da6 P b5715f9d837245fea4eddeac16b3de43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:33:21.001104 18067 raft_consensus.cc:504] T 09188ad29b434082950b5278e4ed0da6 P b5715f9d837245fea4eddeac16b3de43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b447e39c7f794753b08905085d1d4fa2" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 40941 } } peers { permanent_uuid: "1a6f2a62b2ba4b348ac425d72b7d46b1" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39117 } } peers { permanent_uuid: "b5715f9d837245fea4eddeac16b3de43" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 37009 } }
I20260928 06:33:21.001649 18067 leader_election.cc:288] T 09188ad29b434082950b5278e4ed0da6 P b5715f9d837245fea4eddeac16b3de43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b447e39c7f794753b08905085d1d4fa2 (127.16.157.65:40941), 1a6f2a62b2ba4b348ac425d72b7d46b1 (127.16.157.67:39117)
I20260928 06:33:21.005172 18056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:21.002044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.66:60499 (local address 127.16.157.67:39117)
0928 06:33:21.002119 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:21.002121 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:21.002195 (+ 74us) server_negotiation.cc:407] Connection header received
0928 06:33:21.002274 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:21.002275 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:21.002286 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:21.002397 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:33:21.002548 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:21.003253 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:21.004318 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:21.004827 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:21.004912 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:21.004922 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:21.005067 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 06:33:21.005103 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 06:33:21.005187 18068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:21.001838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.67:39117 (local address 127.16.157.66:60499)
0928 06:33:21.002146 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:21.002156 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:21.002204 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:21.002403 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:21.002404 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:21.002405 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:33:21.002479 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:21.002481 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:21.003341 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:21.003342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:21.004216 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:21.004219 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:21.004911 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:21.004912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:21.004990 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:21.004994 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:33:21.005061 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 06:33:21.005102 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":21,"thread_start_us":104,"threads_started":1}
I20260928 06:33:21.005635 17996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09188ad29b434082950b5278e4ed0da6" candidate_uuid: "b5715f9d837245fea4eddeac16b3de43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a6f2a62b2ba4b348ac425d72b7d46b1" is_pre_election: true
I20260928 06:33:21.005788 17996 raft_consensus.cc:2454] T 09188ad29b434082950b5278e4ed0da6 P 1a6f2a62b2ba4b348ac425d72b7d46b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5715f9d837245fea4eddeac16b3de43 in term 0.
I20260928 06:33:21.005745 17910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:21.001826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.65:40941 (local address 127.16.157.66:32901)
0928 06:33:21.001921 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:21.001933 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:21.001987 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:21.002295 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:21.002296 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:21.002307 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:33:21.002386 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:21.002389 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:21.003313 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:21.003314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:21.004640 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:21.004643 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:21.005465 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:21.005466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:21.005547 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:21.005551 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:33:21.005639 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 06:33:21.005683 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76,"mutex_wait_us":22}
I20260928 06:33:21.005869 18058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:21.002063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.66:32901 (local address 127.16.157.65:40941)
0928 06:33:21.002125 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:21.002127 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:21.002136 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:33:21.002163 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:21.002164 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:21.002175 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:21.002236 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:33:21.002470 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:21.003219 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:21.004744 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:21.005359 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:21.005467 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:21.005476 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:21.005719 (+ 243us) server_negotiation.cc:299] Negotiation successful
0928 06:33:21.005771 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 06:33:21.006027 17796 leader_election.cc:302] T 09188ad29b434082950b5278e4ed0da6 P b5715f9d837245fea4eddeac16b3de43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a6f2a62b2ba4b348ac425d72b7d46b1, b5715f9d837245fea4eddeac16b3de43; no voters:
I20260928 06:33:21.006210 18067 raft_consensus.cc:2781] T 09188ad29b434082950b5278e4ed0da6 P b5715f9d837245fea4eddeac16b3de43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:33:21.006286 18067 raft_consensus.cc:482] T 09188ad29b434082950b5278e4ed0da6 P b5715f9d837245fea4eddeac16b3de43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:33:21.006314 18067 raft_consensus.cc:3037] T 09188ad29b434082950b5278e4ed0da6 P b5715f9d837245fea4eddeac16b3de43 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:21.006279 17728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09188ad29b434082950b5278e4ed0da6" candidate_uuid: "b5715f9d837245fea4eddeac16b3de43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b447e39c7f794753b08905085d1d4fa2" is_pre_election: true
I20260928 06:33:21.006426 17728 raft_consensus.cc:2454] T 09188ad29b434082950b5278e4ed0da6 P b447e39c7f794753b08905085d1d4fa2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5715f9d837245fea4eddeac16b3de43 in term 0.
I20260928 06:33:21.007491 18067 raft_consensus.cc:504] T 09188ad29b434082950b5278e4ed0da6 P b5715f9d837245fea4eddeac16b3de43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b447e39c7f794753b08905085d1d4fa2" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 40941 } } peers { permanent_uuid: "1a6f2a62b2ba4b348ac425d72b7d46b1" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39117 } } peers { permanent_uuid: "b5715f9d837245fea4eddeac16b3de43" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 37009 } }
I20260928 06:33:21.007725 18067 leader_election.cc:288] T 09188ad29b434082950b5278e4ed0da6 P b5715f9d837245fea4eddeac16b3de43 [CANDIDATE]: Term 1 election: Requested vote from peers b447e39c7f794753b08905085d1d4fa2 (127.16.157.65:40941), 1a6f2a62b2ba4b348ac425d72b7d46b1 (127.16.157.67:39117)
I20260928 06:33:21.007786 17728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09188ad29b434082950b5278e4ed0da6" candidate_uuid: "b5715f9d837245fea4eddeac16b3de43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b447e39c7f794753b08905085d1d4fa2"
I20260928 06:33:21.007908 17996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09188ad29b434082950b5278e4ed0da6" candidate_uuid: "b5715f9d837245fea4eddeac16b3de43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a6f2a62b2ba4b348ac425d72b7d46b1"
I20260928 06:33:21.007897 17728 raft_consensus.cc:3037] T 09188ad29b434082950b5278e4ed0da6 P b447e39c7f794753b08905085d1d4fa2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:21.007974 17996 raft_consensus.cc:3037] T 09188ad29b434082950b5278e4ed0da6 P 1a6f2a62b2ba4b348ac425d72b7d46b1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:21.008826 17728 raft_consensus.cc:2454] T 09188ad29b434082950b5278e4ed0da6 P b447e39c7f794753b08905085d1d4fa2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5715f9d837245fea4eddeac16b3de43 in term 1.
I20260928 06:33:21.008836 17996 raft_consensus.cc:2454] T 09188ad29b434082950b5278e4ed0da6 P 1a6f2a62b2ba4b348ac425d72b7d46b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5715f9d837245fea4eddeac16b3de43 in term 1.
I20260928 06:33:21.009008 17796 leader_election.cc:302] T 09188ad29b434082950b5278e4ed0da6 P b5715f9d837245fea4eddeac16b3de43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a6f2a62b2ba4b348ac425d72b7d46b1, b5715f9d837245fea4eddeac16b3de43; no voters:
I20260928 06:33:21.009133 18067 raft_consensus.cc:2781] T 09188ad29b434082950b5278e4ed0da6 P b5715f9d837245fea4eddeac16b3de43 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:33:21.009296 18067 raft_consensus.cc:686] T 09188ad29b434082950b5278e4ed0da6 P b5715f9d837245fea4eddeac16b3de43 [term 1 LEADER]: Becoming Leader. State: Replica: b5715f9d837245fea4eddeac16b3de43, State: Running, Role: LEADER
I20260928 06:33:21.009577 18067 consensus_queue.cc:237] T 09188ad29b434082950b5278e4ed0da6 P b5715f9d837245fea4eddeac16b3de43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b447e39c7f794753b08905085d1d4fa2" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 40941 } } peers { permanent_uuid: "1a6f2a62b2ba4b348ac425d72b7d46b1" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39117 } } peers { permanent_uuid: "b5715f9d837245fea4eddeac16b3de43" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 37009 } }
I20260928 06:33:21.010375 17583 catalog_manager.cc:5495] T 09188ad29b434082950b5278e4ed0da6 P b5715f9d837245fea4eddeac16b3de43 reported cstate change: term changed from 0 to 1, leader changed from <none> to b5715f9d837245fea4eddeac16b3de43 (127.16.157.66). New cstate: current_term: 1 leader_uuid: "b5715f9d837245fea4eddeac16b3de43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b447e39c7f794753b08905085d1d4fa2" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 40941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1a6f2a62b2ba4b348ac425d72b7d46b1" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5715f9d837245fea4eddeac16b3de43" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 37009 } health_report { overall_health: HEALTHY } } }
Sep 28 06:33:21 dist-test-slave-rrj5 krb5kdc[17538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.157.65@KRBTEST.COM
I20260928 06:33:21.030376 18058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:21.021483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34210 (local address 127.16.157.65:40941)
0928 06:33:21.021538 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:21.021541 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:21.021610 (+ 69us) server_negotiation.cc:407] Connection header received
0928 06:33:21.021724 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:21.021725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:21.021736 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:21.021812 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:21.022599 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:21.023091 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:21.024260 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:21.024437 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:21.024484 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:21.027160 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:21.027170 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:21.027171 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:21.027179 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:21.029127 (+ 1948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:21.029454 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:21.029458 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:21.029458 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:21.029528 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:21.029765 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:21.029766 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:21.029766 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:21.029958 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:21.030118 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:21.030159 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 06:33:21.030258 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
W20260928 06:33:21.039844 17775 tablet.cc:2369] T 09188ad29b434082950b5278e4ed0da6 P b447e39c7f794753b08905085d1d4fa2: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 06:33:21 dist-test-slave-rrj5 krb5kdc[17538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577201, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 06:33:21 dist-test-slave-rrj5 krb5kdc[17538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577201, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.157.65@KRBTEST.COM
I20260928 06:33:21.054244 18058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:21.046877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34218 (local address 127.16.157.65:40941)
0928 06:33:21.046941 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:21.046943 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:21.047025 (+ 82us) server_negotiation.cc:407] Connection header received
0928 06:33:21.047121 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:21.047122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:21.047132 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:21.047184 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:21.047737 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:21.048218 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:21.048944 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:21.049145 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:21.049222 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:21.051545 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:21.051547 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:21.051547 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:21.051555 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:21.052955 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:21.053335 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:21.053337 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:21.053337 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:21.053397 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:21.053637 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:21.053638 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:21.053638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:21.053748 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:21.053873 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:21.054059 (+ 186us) server_negotiation.cc:299] Negotiation successful
0928 06:33:21.054142 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
Sep 28 06:33:21 dist-test-slave-rrj5 krb5kdc[17538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 06:33:21 dist-test-slave-rrj5 krb5kdc[17538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.157.65@KRBTEST.COM
I20260928 06:33:21.077872 18058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:21.069838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34230 (local address 127.16.157.65:40941)
0928 06:33:21.069912 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:21.069918 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:21.070006 (+ 88us) server_negotiation.cc:407] Connection header received
0928 06:33:21.070082 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:21.070083 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:21.070092 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:21.070143 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:21.070900 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:21.071372 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:21.072255 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:21.072452 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:21.072528 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:21.075124 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:21.075126 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:21.075127 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:21.075135 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:21.076734 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:21.077076 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:21.077078 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:21.077078 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:21.077138 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:21.077342 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:21.077343 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:21.077344 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:21.077444 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:21.077546 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:21.077695 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 06:33:21.077773 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260928 06:33:21.078348 17688 tablet_service.cc:2908] Rejecting scan request for tablet 09188ad29b434082950b5278e4ed0da6: Uninitialized: clean time has not yet been initialized
I20260928 06:33:21.079264 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 17645
I20260928 06:33:21.085820 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 17779
I20260928 06:33:21.091966 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 17913
I20260928 06:33:21.096427 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 17551
2026-09-28T06:33:21Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2112 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:33:21 dist-test-slave-rrj5 krb5kdc[18083](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:33:21 dist-test-slave-rrj5 krb5kdc[18083](info): set up 2 sockets
Sep 28 06:33:21 dist-test-slave-rrj5 krb5kdc[18083](info): commencing operation
krb5kdc: starting...
W20260928 06:33:22.203570 17013 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Sep 28 06:33:22 dist-test-slave-rrj5 krb5kdc[18083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577202, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:33:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:33:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.157.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab.
Entry for principal kudu/127.16.157.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab.
I20260928 06:33:22.327853 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:40707
--webserver_interface=127.16.157.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=kudu/127.16.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:38327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.157.126:40707
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:22.418885 18096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:22.419168 18096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:22.419212 18096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:22.420501 18096 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:33:22.420547 18096 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 06:33:22.420591 18096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:22.420617 18096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:22.420652 18096 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:33:22.420684 18096 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:33:22.422380 18096 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:38327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.157.126:40707
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=kudu/127.16.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:40707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:22.422660 18096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:22.422891 18096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:22.427875 18102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:22.427920 18096 server_base.cc:1034] running on GCE node
W20260928 06:33:22.427893 18101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:22.428125 18104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:22.428395 18096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:22.428732 18096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:22.429906 18096 hybrid_clock.cc:648] HybridClock initialized: now 1790577202429889 us; error 34 us; skew 500 ppm
Sep 28 06:33:22 dist-test-slave-rrj5 krb5kdc[18083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577202, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:22.435954 18096 init.cc:377] Logged in from keytab as kudu/127.16.157.126@KRBTEST.COM (short username kudu)
I20260928 06:33:22.436390 18096 webserver.cc:460] Webserver started at http://127.16.157.126:37319/ using document root <none> and password file <none>
I20260928 06:33:22.436625 18096 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:22.436693 18096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:22.436869 18096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:22.438099 18096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/master-0/data/instance:
uuid: "ee1eda9c7d684f88a77cb007b5502e30"
format_stamp: "Formatted at 2026-09-28 06:33:22 on dist-test-slave-rrj5"
server_key: "abc8d31213c89d436a4b10c601789d6c"
server_key_iv: "5d23213019eca82b79e3719212193045"
server_key_version: "encryptionkey@0"
I20260928 06:33:22.438475 18096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/master-0/wal/instance:
uuid: "ee1eda9c7d684f88a77cb007b5502e30"
format_stamp: "Formatted at 2026-09-28 06:33:22 on dist-test-slave-rrj5"
server_key: "abc8d31213c89d436a4b10c601789d6c"
server_key_iv: "5d23213019eca82b79e3719212193045"
server_key_version: "encryptionkey@0"
I20260928 06:33:22.439927 18096 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:33:22.440752 18111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:22.440954 18096 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:22.441056 18096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/master-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/master-0/wal
uuid: "ee1eda9c7d684f88a77cb007b5502e30"
format_stamp: "Formatted at 2026-09-28 06:33:22 on dist-test-slave-rrj5"
server_key: "abc8d31213c89d436a4b10c601789d6c"
server_key_iv: "5d23213019eca82b79e3719212193045"
server_key_version: "encryptionkey@0"
I20260928 06:33:22.441139 18096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:22.469067 18096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:22.471357 18096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:22.471516 18096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:22.476374 18096 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.126:40707
I20260928 06:33:22.476491 18173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.126:40707 every 8 connection(s)
I20260928 06:33:22.476768 18096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
I20260928 06:33:22.477542 18174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:22.481082 18174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee1eda9c7d684f88a77cb007b5502e30: Bootstrap starting.
I20260928 06:33:22.481738 18174 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee1eda9c7d684f88a77cb007b5502e30: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:22.482033 18174 log.cc:824] T 00000000000000000000000000000000 P ee1eda9c7d684f88a77cb007b5502e30: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:22.482834 18174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee1eda9c7d684f88a77cb007b5502e30: No bootstrap required, opened a new log
I20260928 06:33:22.483171 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 18096
I20260928 06:33:22.483268 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/master-0/wal/instance
I20260928 06:33:22.483429 17013 external_mini_cluster.cc:1428] Setting key 81e2f93839e2b76940613aec2b52b746
I20260928 06:33:22.485759 18174 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee1eda9c7d684f88a77cb007b5502e30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee1eda9c7d684f88a77cb007b5502e30" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40707 } }
I20260928 06:33:22.485919 18174 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee1eda9c7d684f88a77cb007b5502e30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:22.485965 18174 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee1eda9c7d684f88a77cb007b5502e30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee1eda9c7d684f88a77cb007b5502e30, State: Initialized, Role: FOLLOWER
I20260928 06:33:22.486099 18174 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee1eda9c7d684f88a77cb007b5502e30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee1eda9c7d684f88a77cb007b5502e30" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40707 } }
I20260928 06:33:22.486195 18174 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee1eda9c7d684f88a77cb007b5502e30 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:33:22.486234 18174 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee1eda9c7d684f88a77cb007b5502e30 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:33:22.486297 18174 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee1eda9c7d684f88a77cb007b5502e30 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:22.486864 18174 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee1eda9c7d684f88a77cb007b5502e30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee1eda9c7d684f88a77cb007b5502e30" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40707 } }
I20260928 06:33:22.486994 18174 leader_election.cc:302] T 00000000000000000000000000000000 P ee1eda9c7d684f88a77cb007b5502e30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee1eda9c7d684f88a77cb007b5502e30; no voters:
I20260928 06:33:22.487164 18174 leader_election.cc:288] T 00000000000000000000000000000000 P ee1eda9c7d684f88a77cb007b5502e30 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:33:22.487205 18179 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee1eda9c7d684f88a77cb007b5502e30 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:33:22.487356 18179 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee1eda9c7d684f88a77cb007b5502e30 [term 1 LEADER]: Becoming Leader. State: Replica: ee1eda9c7d684f88a77cb007b5502e30, State: Running, Role: LEADER
I20260928 06:33:22.487419 18174 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee1eda9c7d684f88a77cb007b5502e30 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:33:22.487437 18179 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee1eda9c7d684f88a77cb007b5502e30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee1eda9c7d684f88a77cb007b5502e30" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40707 } }
I20260928 06:33:22.487744 18179 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee1eda9c7d684f88a77cb007b5502e30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee1eda9c7d684f88a77cb007b5502e30. Latest consensus state: current_term: 1 leader_uuid: "ee1eda9c7d684f88a77cb007b5502e30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee1eda9c7d684f88a77cb007b5502e30" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40707 } } }
I20260928 06:33:22.487753 18180 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee1eda9c7d684f88a77cb007b5502e30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee1eda9c7d684f88a77cb007b5502e30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee1eda9c7d684f88a77cb007b5502e30" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40707 } } }
I20260928 06:33:22.487830 18179 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee1eda9c7d684f88a77cb007b5502e30 [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:22.487838 18180 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee1eda9c7d684f88a77cb007b5502e30 [sys.catalog]: This master's current role is: LEADER
W20260928 06:33:22.489172 18195 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ee1eda9c7d684f88a77cb007b5502e30: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:33:22.489740 18195 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:33:22.490645 18185 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:33:22.490870 18185 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 06:33:22 dist-test-slave-rrj5 krb5kdc[18083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577202, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:33:22.493383 18185 catalog_manager.cc:1347] Generated new cluster ID: 245dda539b3d4b8b941b8b0cf37e87df
I20260928 06:33:22.493446 18185 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:33:22.498284 18178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:22.483997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33236 (local address 127.16.157.126:40707)
0928 06:33:22.484156 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:22.484165 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:22.484198 (+ 33us) server_negotiation.cc:407] Connection header received
0928 06:33:22.485757 (+ 1559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:22.485768 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:22.485848 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:22.486018 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:22.487024 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:22.488690 (+ 1666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:22.489665 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:22.490412 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:22.490547 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:22.493352 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:22.493357 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:22.493357 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:22.493369 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:22.496071 (+ 2702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:22.496490 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:22.496493 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:22.496493 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:22.496634 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:22.496933 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:22.496933 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:22.496934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:22.497241 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:22.497859 (+ 618us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:22.497934 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 06:33:22.498075 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":97,"threads_started":1}
I20260928 06:33:22.504786 18185 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:33:22.505558 18185 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:33:22.512256 18185 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee1eda9c7d684f88a77cb007b5502e30: Generated new TSK 0
I20260928 06:33:22.512445 18185 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.157.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab.
Entry for principal kudu/127.16.157.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab.
I20260928 06:33:22.544340 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.65:0
--local_ip_for_outbound_sockets=127.16.157.65
--webserver_interface=127.16.157.65
--webserver_port=0
--tserver_master_addrs=127.16.157.126:40707
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab
--principal=kudu/127.16.157.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:38327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:22.646615 18200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:22.646835 18200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:22.646876 18200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:22.647970 18200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:22.648044 18200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:22.648080 18200 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:33:22.648109 18200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.65
I20260928 06:33:22.649734 18200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:38327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab
--principal=kudu/127.16.157.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.157.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.157.126:40707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.65
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:22.650000 18200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:22.650234 18200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:22.655797 18205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:22.655820 18206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:22.655902 18200 server_base.cc:1034] running on GCE node
W20260928 06:33:22.656026 18208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:22.656243 18200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:22.656488 18200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:22.657655 18200 hybrid_clock.cc:648] HybridClock initialized: now 1790577202657630 us; error 41 us; skew 500 ppm
Sep 28 06:33:22 dist-test-slave-rrj5 krb5kdc[18083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577202, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:22.660099 18200 init.cc:377] Logged in from keytab as kudu/127.16.157.65@KRBTEST.COM (short username kudu)
I20260928 06:33:22.660526 18200 webserver.cc:460] Webserver started at http://127.16.157.65:40741/ using document root <none> and password file <none>
I20260928 06:33:22.660758 18200 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:22.660873 18200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:22.661026 18200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:22.662147 18200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/ts-0/data/instance:
uuid: "8cf20e908f614fea9452dbe848d1dac3"
format_stamp: "Formatted at 2026-09-28 06:33:22 on dist-test-slave-rrj5"
server_key: "3f814179419320da886b5d0d1fb32806"
server_key_iv: "c014d490b26af1c8dbfec23507b65134"
server_key_version: "encryptionkey@0"
I20260928 06:33:22.662508 18200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/ts-0/wal/instance:
uuid: "8cf20e908f614fea9452dbe848d1dac3"
format_stamp: "Formatted at 2026-09-28 06:33:22 on dist-test-slave-rrj5"
server_key: "3f814179419320da886b5d0d1fb32806"
server_key_iv: "c014d490b26af1c8dbfec23507b65134"
server_key_version: "encryptionkey@0"
I20260928 06:33:22.663846 18200 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:33:22.664727 18215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:22.664898 18200 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:22.665001 18200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/ts-0/wal
uuid: "8cf20e908f614fea9452dbe848d1dac3"
format_stamp: "Formatted at 2026-09-28 06:33:22 on dist-test-slave-rrj5"
server_key: "3f814179419320da886b5d0d1fb32806"
server_key_iv: "c014d490b26af1c8dbfec23507b65134"
server_key_version: "encryptionkey@0"
I20260928 06:33:22.665071 18200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:22.696501 18200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:22.699034 18200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:22.699213 18200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:22.699870 18200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:33:22.699924 18200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:22.699957 18200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:33:22.699983 18200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 06:33:22 dist-test-slave-rrj5 krb5kdc[18083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577202, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.65@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:33:22.707823 18200 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.65:35783
I20260928 06:33:22.708231 18200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
I20260928 06:33:22.709108 18330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.65:35783 every 8 connection(s)
I20260928 06:33:22.710316 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 18200
I20260928 06:33:22.710418 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/ts-0/wal/instance
I20260928 06:33:22.710585 17013 external_mini_cluster.cc:1428] Setting key 15ab6b536bb90af0a24177273599022c
I20260928 06:33:22.710594 18178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:22.700260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:35365 (local address 127.16.157.126:40707)
0928 06:33:22.700310 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:22.700312 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:22.700569 (+ 257us) server_negotiation.cc:407] Connection header received
0928 06:33:22.701104 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:22.701105 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:22.701118 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:22.701169 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:22.702010 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:22.702827 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:22.704351 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:22.704690 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:22.704763 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:22.707323 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:22.707325 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:22.707325 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:22.707337 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:22.708925 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:22.709427 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:22.709428 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:22.709428 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:22.709496 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:22.709813 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:22.709814 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:22.709814 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:22.709974 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:22.710057 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:22.710406 (+ 349us) server_negotiation.cc:299] Negotiation successful
0928 06:33:22.710498 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:33:22.710742 18224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:22.700265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40707 (local address 127.16.157.65:35365)
0928 06:33:22.700428 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:22.700467 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:22.700978 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:22.701254 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:22.701257 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:22.701572 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:22.701887 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:22.701893 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:22.702922 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:22.702923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:22.704242 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:22.704246 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:22.704782 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:22.704783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:22.704931 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:22.705469 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:22.705479 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:22.707240 (+ 1761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:22.709019 (+ 1779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:22.709025 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:22.709027 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:22.709343 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:22.709615 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:22.709616 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:22.709616 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:22.709731 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:22.710049 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:22.710053 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:22.710222 (+ 169us) client_negotiation.cc:769] Sending connection context
0928 06:33:22.710338 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 06:33:22.710511 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":89,"threads_started":1}
I20260928 06:33:22.711584 18331 heartbeater.cc:347] Connected to a master server at 127.16.157.126:40707
I20260928 06:33:22.711669 18331 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:22.711859 18331 heartbeater.cc:511] Master 127.16.157.126:40707 requested a full tablet report, sending...
I20260928 06:33:22.712374 18126 ts_manager.cc:194] Registered new tserver with Master: 8cf20e908f614fea9452dbe848d1dac3 (127.16.157.65:35783)
I20260928 06:33:22.713289 18126 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.157.65@KRBTEST.COM'} at 127.16.157.65:35365
I20260928 06:33:22.719681 18337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:22.712676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40707 (local address 127.16.157.65:35025)
0928 06:33:22.712803 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:22.712817 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:22.712873 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:22.714292 (+ 1419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:22.714293 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:22.714444 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:22.714492 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:22.714493 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:22.715151 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:22.715152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:22.715730 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:22.715732 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:22.716138 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:22.716139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:22.716213 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:22.716637 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:22.716649 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:22.716916 (+ 267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:22.718621 (+ 1705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:22.718622 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:22.718623 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:22.718800 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:22.719026 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:22.719027 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:22.719027 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:22.719090 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:22.719341 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:22.719342 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:22.719399 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 06:33:22.719481 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 06:33:22.719586 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":72,"threads_started":1}
I20260928 06:33:22.719681 18178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:22.713501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:35025 (local address 127.16.157.126:40707)
0928 06:33:22.713538 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:22.713540 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:22.713550 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:33:22.713591 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:22.713591 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:22.713601 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:22.713662 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:22.714596 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:22.715058 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:22.715839 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:22.716051 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:22.716127 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:22.717027 (+ 900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:22.717029 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:22.717029 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:22.717036 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:22.718520 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:22.718874 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:22.718875 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:22.718875 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:22.718928 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:22.719154 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:22.719155 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:22.719155 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:22.719265 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:22.719348 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:22.719519 (+ 171us) server_negotiation.cc:299] Negotiation successful
0928 06:33:22.719599 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
WARNING: no policy specified for kudu/127.16.157.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab.
Entry for principal kudu/127.16.157.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab.
I20260928 06:33:22.738485 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.66:0
--local_ip_for_outbound_sockets=127.16.157.66
--webserver_interface=127.16.157.66
--webserver_port=0
--tserver_master_addrs=127.16.157.126:40707
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab
--principal=kudu/127.16.157.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:38327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:22.824411 18340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:22.824653 18340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:22.824690 18340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:22.825947 18340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:22.826006 18340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:22.826043 18340 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:33:22.826093 18340 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.66
I20260928 06:33:22.827468 18340 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:38327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab
--principal=kudu/127.16.157.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.157.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.157.126:40707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18340
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.66
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:22.827715 18340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:22.827930 18340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:22.832839 18345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:22.832818 18348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:22.833014 18340 server_base.cc:1034] running on GCE node
W20260928 06:33:22.832818 18346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:22.833328 18340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:22.833644 18340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:22.834823 18340 hybrid_clock.cc:648] HybridClock initialized: now 1790577202834810 us; error 31 us; skew 500 ppm
Sep 28 06:33:22 dist-test-slave-rrj5 krb5kdc[18083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577202, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:22.837729 18340 init.cc:377] Logged in from keytab as kudu/127.16.157.66@KRBTEST.COM (short username kudu)
I20260928 06:33:22.838268 18340 webserver.cc:460] Webserver started at http://127.16.157.66:36699/ using document root <none> and password file <none>
I20260928 06:33:22.838546 18340 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:22.838619 18340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:22.838760 18340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:22.840165 18340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/ts-1/data/instance:
uuid: "02a32df533b64624934cc9f57e59fc8b"
format_stamp: "Formatted at 2026-09-28 06:33:22 on dist-test-slave-rrj5"
server_key: "ebf0883b7e0e20ee031c7085c7a3d9d7"
server_key_iv: "ced2dc8c5d4509f5c3a38b1aafe349d9"
server_key_version: "encryptionkey@0"
I20260928 06:33:22.840605 18340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/ts-1/wal/instance:
uuid: "02a32df533b64624934cc9f57e59fc8b"
format_stamp: "Formatted at 2026-09-28 06:33:22 on dist-test-slave-rrj5"
server_key: "ebf0883b7e0e20ee031c7085c7a3d9d7"
server_key_iv: "ced2dc8c5d4509f5c3a38b1aafe349d9"
server_key_version: "encryptionkey@0"
I20260928 06:33:22.842376 18340 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260928 06:33:22.843251 18355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:22.843498 18340 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:22.843604 18340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/ts-1/wal
uuid: "02a32df533b64624934cc9f57e59fc8b"
format_stamp: "Formatted at 2026-09-28 06:33:22 on dist-test-slave-rrj5"
server_key: "ebf0883b7e0e20ee031c7085c7a3d9d7"
server_key_iv: "ced2dc8c5d4509f5c3a38b1aafe349d9"
server_key_version: "encryptionkey@0"
I20260928 06:33:22.843677 18340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:22.859622 18340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:22.861965 18340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:22.862113 18340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:22.862660 18340 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:33:22.862702 18340 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:22.862746 18340 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:33:22.862773 18340 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 06:33:22 dist-test-slave-rrj5 krb5kdc[18083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577202, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.66@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:33:22.870981 18340 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.66:33503
I20260928 06:33:22.871057 18470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.66:33503 every 8 connection(s)
I20260928 06:33:22.871411 18340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
I20260928 06:33:22.873301 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 18340
I20260928 06:33:22.873449 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/ts-1/wal/instance
I20260928 06:33:22.873711 17013 external_mini_cluster.cc:1428] Setting key c1daa21154240ac429365aafed89f3fd
I20260928 06:33:22.875284 18364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:22.863213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40707 (local address 127.16.157.66:49521)
0928 06:33:22.863402 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:22.863450 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:22.864103 (+ 653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:22.864412 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:22.864418 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:22.864775 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:22.865040 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:22.865046 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:22.866096 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:22.866097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:22.867412 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:22.867416 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:22.867980 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:22.867982 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:22.868140 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:22.868804 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:22.868816 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:22.871531 (+ 2715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:22.873654 (+ 2123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:22.873661 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:22.873663 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:22.873961 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:22.874226 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:22.874227 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:22.874227 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:22.874355 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:22.874636 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:22.874642 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:22.874788 (+ 146us) client_negotiation.cc:769] Sending connection context
0928 06:33:22.874891 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 06:33:22.875052 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":104,"threads_started":1}
I20260928 06:33:22.875802 18178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:22.863231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.66:49521 (local address 127.16.157.126:40707)
0928 06:33:22.863279 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:22.863281 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:22.863558 (+ 277us) server_negotiation.cc:407] Connection header received
0928 06:33:22.864242 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:22.864244 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:22.864260 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:22.864342 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:22.865131 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:22.866000 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:22.867525 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:22.867823 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:22.867894 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:22.871714 (+ 3820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:22.871715 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:22.871715 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:22.871725 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:22.873505 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:22.874081 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:22.874082 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:22.874082 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:22.874142 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:22.874425 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:22.874426 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:22.874426 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:22.874554 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:22.874612 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:22.875623 (+ 1011us) server_negotiation.cc:299] Negotiation successful
0928 06:33:22.875708 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 06:33:22.877377 18471 heartbeater.cc:347] Connected to a master server at 127.16.157.126:40707
I20260928 06:33:22.877504 18471 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:22.877732 18471 heartbeater.cc:511] Master 127.16.157.126:40707 requested a full tablet report, sending...
I20260928 06:33:22.878118 18126 ts_manager.cc:194] Registered new tserver with Master: 02a32df533b64624934cc9f57e59fc8b (127.16.157.66:33503)
I20260928 06:33:22.878626 18126 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.157.66@KRBTEST.COM'} at 127.16.157.66:49521
WARNING: no policy specified for kudu/127.16.157.67@KRBTEST.COM; defaulting to no policy
I20260928 06:33:22.884531 18178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:22.877515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.66:57009 (local address 127.16.157.126:40707)
0928 06:33:22.877554 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:22.877556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:22.877809 (+ 253us) server_negotiation.cc:407] Connection header received
0928 06:33:22.877884 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:22.877885 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:22.877896 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:22.877959 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:22.878319 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:22.879093 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:22.880097 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:22.880378 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:22.880414 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:22.881689 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:22.881690 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:22.881691 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:22.881697 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:22.883303 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:22.883695 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:22.883696 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:22.883696 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:22.883755 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:22.883977 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:22.883978 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:22.883978 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:22.884155 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:22.884221 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:22.884377 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 06:33:22.884445 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 06:33:22.884546 18478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:22.877607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40707 (local address 127.16.157.66:57009)
0928 06:33:22.877742 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:22.877755 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:22.877822 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:22.877964 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:22.877965 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:22.878172 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:22.878234 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:22.878236 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:22.879195 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:22.879197 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:22.879990 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:22.879993 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:22.880532 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:22.880534 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:22.880608 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:22.881091 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:22.881098 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:22.881565 (+ 467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:22.883398 (+ 1833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:22.883399 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:22.883400 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:22.883607 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:22.883843 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:22.883844 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:22.883844 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:22.883911 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:22.884216 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:22.884217 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:22.884266 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 06:33:22.884344 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 06:33:22.884447 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":72,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab.
Entry for principal kudu/127.16.157.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab.
I20260928 06:33:22.904191 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.67:0
--local_ip_for_outbound_sockets=127.16.157.67
--webserver_interface=127.16.157.67
--webserver_port=0
--tserver_master_addrs=127.16.157.126:40707
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab
--principal=kudu/127.16.157.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:38327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:23.008692 18480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:23.008910 18480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:23.008953 18480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:23.010206 18480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:23.010246 18480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:23.010267 18480 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:33:23.010284 18480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.67
I20260928 06:33:23.011826 18480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:38327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab
--principal=kudu/127.16.157.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.157.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.157.126:40707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.67
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:23.012115 18480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:23.012378 18480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:23.017503 18486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:23.017539 18485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:23.017547 18488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:23.017930 18480 server_base.cc:1034] running on GCE node
I20260928 06:33:23.018143 18480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:23.018388 18480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:23.019555 18480 hybrid_clock.cc:648] HybridClock initialized: now 1790577203019537 us; error 29 us; skew 500 ppm
Sep 28 06:33:23 dist-test-slave-rrj5 krb5kdc[18083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577203, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:23.022627 18480 init.cc:377] Logged in from keytab as kudu/127.16.157.67@KRBTEST.COM (short username kudu)
I20260928 06:33:23.023037 18480 webserver.cc:460] Webserver started at http://127.16.157.67:42607/ using document root <none> and password file <none>
I20260928 06:33:23.023289 18480 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:23.023358 18480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:23.023505 18480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:23.024488 18480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/ts-2/data/instance:
uuid: "a5ca55a791204376a9ddd1bd72b4cff6"
format_stamp: "Formatted at 2026-09-28 06:33:23 on dist-test-slave-rrj5"
server_key: "d4dc7789549f6c3ba3aabfce34683532"
server_key_iv: "8f8cad7fef8f3341ffbdab28c6c63963"
server_key_version: "encryptionkey@0"
I20260928 06:33:23.024900 18480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/ts-2/wal/instance:
uuid: "a5ca55a791204376a9ddd1bd72b4cff6"
format_stamp: "Formatted at 2026-09-28 06:33:23 on dist-test-slave-rrj5"
server_key: "d4dc7789549f6c3ba3aabfce34683532"
server_key_iv: "8f8cad7fef8f3341ffbdab28c6c63963"
server_key_version: "encryptionkey@0"
I20260928 06:33:23.026393 18480 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:33:23.027231 18495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:23.027426 18480 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:33:23.027532 18480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/ts-2/wal
uuid: "a5ca55a791204376a9ddd1bd72b4cff6"
format_stamp: "Formatted at 2026-09-28 06:33:23 on dist-test-slave-rrj5"
server_key: "d4dc7789549f6c3ba3aabfce34683532"
server_key_iv: "8f8cad7fef8f3341ffbdab28c6c63963"
server_key_version: "encryptionkey@0"
I20260928 06:33:23.027640 18480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:23.048378 18480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:23.050700 18480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:23.050887 18480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:23.051476 18480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:33:23.051525 18480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:23.051576 18480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:33:23.051601 18480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 06:33:23 dist-test-slave-rrj5 krb5kdc[18083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577203, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.67@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:33:23.059157 18480 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.67:39679
I20260928 06:33:23.059314 18610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.67:39679 every 8 connection(s)
I20260928 06:33:23.059645 18480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
I20260928 06:33:23.062309 18178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:23.051917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:42347 (local address 127.16.157.126:40707)
0928 06:33:23.051967 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:23.051969 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:23.052395 (+ 426us) server_negotiation.cc:407] Connection header received
0928 06:33:23.053048 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:23.053050 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:23.053064 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:23.053143 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:23.053975 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:23.054809 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:23.056265 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:23.056619 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:23.056700 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:23.059468 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:23.059470 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:23.059470 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:23.059478 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:23.060953 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:23.061416 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:23.061417 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:23.061417 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:23.061468 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:23.061737 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:23.061738 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:23.061738 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:23.061839 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:23.061892 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:23.062180 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 06:33:23.062245 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 06:33:23.062503 18504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:23.051921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40707 (local address 127.16.157.67:42347)
0928 06:33:23.052232 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:23.052282 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:23.052926 (+ 644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:23.053210 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:23.053215 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:23.053552 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:23.053849 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:23.053856 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:23.054928 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:23.054929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:23.056190 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:23.056193 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:23.056788 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:23.056790 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:23.056954 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:23.057501 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:23.057511 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:23.059357 (+ 1846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:23.061037 (+ 1680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:23.061043 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:23.061045 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:23.061338 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:23.061532 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:23.061532 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:23.061533 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:23.061676 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:23.061900 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:23.061904 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:23.062049 (+ 145us) client_negotiation.cc:769] Sending connection context
0928 06:33:23.062144 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 06:33:23.062293 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":91,"threads_started":1}
I20260928 06:33:23.062922 18611 heartbeater.cc:347] Connected to a master server at 127.16.157.126:40707
I20260928 06:33:23.063004 18611 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:23.063164 18611 heartbeater.cc:511] Master 127.16.157.126:40707 requested a full tablet report, sending...
I20260928 06:33:23.063535 18127 ts_manager.cc:194] Registered new tserver with Master: a5ca55a791204376a9ddd1bd72b4cff6 (127.16.157.67:39679)
I20260928 06:33:23.064031 18127 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.157.67@KRBTEST.COM'} at 127.16.157.67:42347
I20260928 06:33:23.069432 18178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:23.064054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:34011 (local address 127.16.157.126:40707)
0928 06:33:23.064195 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:23.064197 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:23.064205 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:33:23.064224 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:23.064224 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:23.064233 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:23.064311 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:23.064622 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:23.065082 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:23.065851 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:23.066048 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:23.066119 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:23.066949 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:23.066950 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:23.066950 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:23.066957 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:23.068331 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:23.068668 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:23.068669 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:23.068670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:23.068723 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:23.068930 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:23.068930 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:23.068931 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:23.069039 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:23.069120 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:23.069270 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 06:33:23.069342 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260928 06:33:23.069447 18617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:23.063988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40707 (local address 127.16.157.67:34011)
0928 06:33:23.064101 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:23.064111 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:23.064161 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:23.064318 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:23.064319 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:23.064495 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:23.064542 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:23.064543 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:23.065172 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:23.065173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:23.065759 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:23.065760 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:23.066119 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:23.066120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:23.066198 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:23.066604 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:23.066608 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:23.066863 (+ 255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:23.068407 (+ 1544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:23.068408 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:23.068409 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:23.068582 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:23.068801 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:23.068802 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:23.068803 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:23.068868 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:23.069112 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:23.069112 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:23.069156 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 06:33:23.069245 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 06:33:23.069344 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":66,"threads_started":1}
I20260928 06:33:23.069850 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 18480
I20260928 06:33:23.069929 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/ts-2/wal/instance
I20260928 06:33:23.070079 17013 external_mini_cluster.cc:1428] Setting key fef65da37eb54611898095e41e421f18
I20260928 06:33:23.071260 17013 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 06:33:23 dist-test-slave-rrj5 krb5kdc[18083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577203, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 06:33:23 dist-test-slave-rrj5 krb5kdc[18083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577203, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:33:23.098446 18178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:23.090649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33252 (local address 127.16.157.126:40707)
0928 06:33:23.090702 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:23.090705 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:23.090825 (+ 120us) server_negotiation.cc:407] Connection header received
0928 06:33:23.090862 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:23.090863 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:23.090874 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:23.090944 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:23.091469 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:23.092318 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:23.093044 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:23.093335 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:23.093407 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:23.095804 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:23.095805 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:23.095805 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:23.095813 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:23.097318 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:23.097674 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:23.097675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:23.097675 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:23.097737 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:23.097931 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:23.097932 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:23.097932 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:23.098064 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:23.098164 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:23.098277 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 06:33:23.098354 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 06:33:23.100250 18127 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33252:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:33:23.102432 18127 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:33:23.109509 18628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:23.104742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.65:35783 (local address 127.0.0.1:41740)
0928 06:33:23.104871 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:23.104908 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:23.104987 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:23.105537 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:23.105540 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:23.105542 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:33:23.105649 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:23.105652 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:23.106972 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:23.106973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:23.108269 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:23.108272 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:23.109221 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:23.109222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:23.109303 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:23.109316 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 06:33:23.109379 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 06:33:23.109428 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"mutex_wait_us":108,"thread_start_us":67,"threads_started":1}
I20260928 06:33:23.109903 18631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:23.104993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41740 (local address 127.16.157.65:35783)
0928 06:33:23.105367 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:23.105371 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:23.105380 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:33:23.105410 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:23.105411 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:23.105424 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:23.105518 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:33:23.105740 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:23.106883 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:23.108359 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:23.109049 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:23.109117 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:23.109172 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:23.109705 (+ 533us) server_negotiation.cc:299] Negotiation successful
0928 06:33:23.109742 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":94,"threads_started":1}
I20260928 06:33:23.109804 18629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:23.104751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.67:39679 (local address 127.0.0.1:36364)
0928 06:33:23.105703 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:23.105714 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:23.105784 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:23.105973 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:23.105973 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:23.105974 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:33:23.106015 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:23.106016 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:23.107186 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:23.107187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:23.108450 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:23.108452 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:23.109353 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:23.109354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:23.109419 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:23.109422 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:33:23.109465 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 06:33:23.109495 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":922,"mutex_wait_us":283,"thread_start_us":86,"threads_started":1}
I20260928 06:33:23.109953 18633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:23.105181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36364 (local address 127.16.157.67:39679)
0928 06:33:23.105351 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:23.105355 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:23.105782 (+ 427us) server_negotiation.cc:407] Connection header received
0928 06:33:23.105853 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:23.105856 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:23.105871 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:23.105963 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:33:23.106105 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:23.107106 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:23.108532 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:23.109214 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:23.109272 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:23.109323 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:23.109547 (+ 224us) server_negotiation.cc:299] Negotiation successful
0928 06:33:23.109857 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":84,"threads_started":1}
I20260928 06:33:23.110707 18545 tablet_service.cc:1467] Processing CreateTablet for tablet 4e3e4c32f4cc472a9f838fcb8d0e3059 (DEFAULT_TABLE table=test-table [id=c661596f0a674ede9551b094579e7502]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:33:23.110771 18265 tablet_service.cc:1467] Processing CreateTablet for tablet 4e3e4c32f4cc472a9f838fcb8d0e3059 (DEFAULT_TABLE table=test-table [id=c661596f0a674ede9551b094579e7502]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:33:23.111066 18545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e3e4c32f4cc472a9f838fcb8d0e3059. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:23.111153 18265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e3e4c32f4cc472a9f838fcb8d0e3059. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:23.113811 18634 tablet_bootstrap.cc:492] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P a5ca55a791204376a9ddd1bd72b4cff6: Bootstrap starting.
I20260928 06:33:23.114609 18634 tablet_bootstrap.cc:654] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P a5ca55a791204376a9ddd1bd72b4cff6: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:23.114933 18634 log.cc:824] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P a5ca55a791204376a9ddd1bd72b4cff6: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:23.115697 18634 tablet_bootstrap.cc:492] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P a5ca55a791204376a9ddd1bd72b4cff6: No bootstrap required, opened a new log
I20260928 06:33:23.115728 18630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:23.104988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.66:33503 (local address 127.0.0.1:54412)
0928 06:33:23.105857 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:23.105866 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:23.105921 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:23.106146 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:23.106147 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:23.106149 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:33:23.106205 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:23.106206 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:23.107423 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:23.107424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:23.108819 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:23.108823 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:23.115476 (+ 6653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:23.115477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:23.115552 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:23.115557 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:33:23.115610 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 06:33:23.115655 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":854,"thread_start_us":79,"threads_started":1}
I20260928 06:33:23.115782 18634 ts_tablet_manager.cc:1397] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P a5ca55a791204376a9ddd1bd72b4cff6: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 06:33:23.116102 18632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:23.105116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54412 (local address 127.16.157.66:33503)
0928 06:33:23.105320 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:23.105326 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:23.105926 (+ 600us) server_negotiation.cc:407] Connection header received
0928 06:33:23.105975 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:23.105977 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:23.105990 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:23.106060 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:33:23.106277 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:23.107324 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:23.108922 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:23.109694 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:23.109775 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:23.115915 (+ 6140us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:23.115943 (+ 28us) server_negotiation.cc:299] Negotiation successful
0928 06:33:23.115991 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":95,"threads_started":1}
I20260928 06:33:23.116711 18405 tablet_service.cc:1467] Processing CreateTablet for tablet 4e3e4c32f4cc472a9f838fcb8d0e3059 (DEFAULT_TABLE table=test-table [id=c661596f0a674ede9551b094579e7502]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:33:23.117029 18405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e3e4c32f4cc472a9f838fcb8d0e3059. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:23.117772 18634 raft_consensus.cc:348] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P a5ca55a791204376a9ddd1bd72b4cff6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cf20e908f614fea9452dbe848d1dac3" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 35783 } } peers { permanent_uuid: "02a32df533b64624934cc9f57e59fc8b" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 33503 } } peers { permanent_uuid: "a5ca55a791204376a9ddd1bd72b4cff6" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39679 } }
I20260928 06:33:23.117928 18634 raft_consensus.cc:374] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P a5ca55a791204376a9ddd1bd72b4cff6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:23.117961 18634 raft_consensus.cc:729] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P a5ca55a791204376a9ddd1bd72b4cff6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5ca55a791204376a9ddd1bd72b4cff6, State: Initialized, Role: FOLLOWER
I20260928 06:33:23.118074 18634 consensus_queue.cc:260] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P a5ca55a791204376a9ddd1bd72b4cff6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cf20e908f614fea9452dbe848d1dac3" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 35783 } } peers { permanent_uuid: "02a32df533b64624934cc9f57e59fc8b" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 33503 } } peers { permanent_uuid: "a5ca55a791204376a9ddd1bd72b4cff6" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39679 } }
I20260928 06:33:23.118353 18634 ts_tablet_manager.cc:1428] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P a5ca55a791204376a9ddd1bd72b4cff6: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260928 06:33:23.118386 18611 heartbeater.cc:503] Master 127.16.157.126:40707 was elected leader, sending a full tablet report...
I20260928 06:33:23.118533 18636 tablet_bootstrap.cc:492] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P 8cf20e908f614fea9452dbe848d1dac3: Bootstrap starting.
I20260928 06:33:23.119146 18636 tablet_bootstrap.cc:654] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P 8cf20e908f614fea9452dbe848d1dac3: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:23.119477 18636 log.cc:824] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P 8cf20e908f614fea9452dbe848d1dac3: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:23.120297 18636 tablet_bootstrap.cc:492] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P 8cf20e908f614fea9452dbe848d1dac3: No bootstrap required, opened a new log
I20260928 06:33:23.120354 18636 ts_tablet_manager.cc:1397] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P 8cf20e908f614fea9452dbe848d1dac3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:33:23.120879 18639 tablet_bootstrap.cc:492] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P 02a32df533b64624934cc9f57e59fc8b: Bootstrap starting.
I20260928 06:33:23.121742 18639 tablet_bootstrap.cc:654] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P 02a32df533b64624934cc9f57e59fc8b: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:23.122110 18639 log.cc:824] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P 02a32df533b64624934cc9f57e59fc8b: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:23.122318 18636 raft_consensus.cc:348] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P 8cf20e908f614fea9452dbe848d1dac3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cf20e908f614fea9452dbe848d1dac3" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 35783 } } peers { permanent_uuid: "02a32df533b64624934cc9f57e59fc8b" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 33503 } } peers { permanent_uuid: "a5ca55a791204376a9ddd1bd72b4cff6" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39679 } }
I20260928 06:33:23.122486 18636 raft_consensus.cc:374] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P 8cf20e908f614fea9452dbe848d1dac3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:23.122526 18636 raft_consensus.cc:729] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P 8cf20e908f614fea9452dbe848d1dac3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cf20e908f614fea9452dbe848d1dac3, State: Initialized, Role: FOLLOWER
I20260928 06:33:23.122654 18636 consensus_queue.cc:260] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P 8cf20e908f614fea9452dbe848d1dac3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cf20e908f614fea9452dbe848d1dac3" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 35783 } } peers { permanent_uuid: "02a32df533b64624934cc9f57e59fc8b" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 33503 } } peers { permanent_uuid: "a5ca55a791204376a9ddd1bd72b4cff6" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39679 } }
I20260928 06:33:23.122925 18639 tablet_bootstrap.cc:492] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P 02a32df533b64624934cc9f57e59fc8b: No bootstrap required, opened a new log
I20260928 06:33:23.122925 18636 ts_tablet_manager.cc:1428] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P 8cf20e908f614fea9452dbe848d1dac3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:33:23.122961 18331 heartbeater.cc:503] Master 127.16.157.126:40707 was elected leader, sending a full tablet report...
I20260928 06:33:23.123000 18639 ts_tablet_manager.cc:1397] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P 02a32df533b64624934cc9f57e59fc8b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:33:23.125164 18639 raft_consensus.cc:348] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P 02a32df533b64624934cc9f57e59fc8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cf20e908f614fea9452dbe848d1dac3" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 35783 } } peers { permanent_uuid: "02a32df533b64624934cc9f57e59fc8b" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 33503 } } peers { permanent_uuid: "a5ca55a791204376a9ddd1bd72b4cff6" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39679 } }
I20260928 06:33:23.125312 18639 raft_consensus.cc:374] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P 02a32df533b64624934cc9f57e59fc8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:23.125349 18639 raft_consensus.cc:729] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P 02a32df533b64624934cc9f57e59fc8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02a32df533b64624934cc9f57e59fc8b, State: Initialized, Role: FOLLOWER
I20260928 06:33:23.125453 18639 consensus_queue.cc:260] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P 02a32df533b64624934cc9f57e59fc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cf20e908f614fea9452dbe848d1dac3" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 35783 } } peers { permanent_uuid: "02a32df533b64624934cc9f57e59fc8b" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 33503 } } peers { permanent_uuid: "a5ca55a791204376a9ddd1bd72b4cff6" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39679 } }
I20260928 06:33:23.125769 18639 ts_tablet_manager.cc:1428] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P 02a32df533b64624934cc9f57e59fc8b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:33:23.125799 18471 heartbeater.cc:503] Master 127.16.157.126:40707 was elected leader, sending a full tablet report...
W20260928 06:33:23.209214 18332 tablet.cc:2369] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P 8cf20e908f614fea9452dbe848d1dac3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:33:23.304267 18642 raft_consensus.cc:482] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P 02a32df533b64624934cc9f57e59fc8b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:33:23.304399 18642 raft_consensus.cc:504] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P 02a32df533b64624934cc9f57e59fc8b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cf20e908f614fea9452dbe848d1dac3" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 35783 } } peers { permanent_uuid: "02a32df533b64624934cc9f57e59fc8b" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 33503 } } peers { permanent_uuid: "a5ca55a791204376a9ddd1bd72b4cff6" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39679 } }
I20260928 06:33:23.304831 18642 leader_election.cc:288] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P 02a32df533b64624934cc9f57e59fc8b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8cf20e908f614fea9452dbe848d1dac3 (127.16.157.65:35783), a5ca55a791204376a9ddd1bd72b4cff6 (127.16.157.67:39679)
I20260928 06:33:23.308760 18364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:23.305046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.67:39679 (local address 127.16.157.66:58221)
0928 06:33:23.305120 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:23.305133 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:23.305191 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:23.305467 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:23.305468 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:23.305470 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:33:23.305545 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:23.305548 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:23.306624 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:23.306625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:23.307696 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:23.307699 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:23.308463 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:23.308464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:23.308556 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:23.308560 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:33:23.308610 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 06:33:23.308655 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 06:33:23.309067 18631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:23.305236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.66:41937 (local address 127.16.157.65:35783)
0928 06:33:23.305284 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:23.305286 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:23.305445 (+ 159us) server_negotiation.cc:407] Connection header received
0928 06:33:23.305528 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:23.305529 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:23.305538 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:23.305628 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:33:23.305803 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:23.306298 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:23.307862 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:23.308633 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:23.308674 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:23.308708 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:23.308948 (+ 240us) server_negotiation.cc:299] Negotiation successful
0928 06:33:23.308997 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 06:33:23.309070 18633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:23.305185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.66:58221 (local address 127.16.157.67:39679)
0928 06:33:23.305293 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:23.305295 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:23.305317 (+ 22us) server_negotiation.cc:407] Connection header received
0928 06:33:23.305346 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:23.305347 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:23.305357 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:23.305427 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:33:23.305672 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:23.306520 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:23.307802 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:23.308354 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:23.308421 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:23.308431 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:23.308963 (+ 532us) server_negotiation.cc:299] Negotiation successful
0928 06:33:23.309005 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260928 06:33:23.309304 18643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:23.305049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.65:35783 (local address 127.16.157.66:41937)
0928 06:33:23.305386 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:23.305396 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:23.305470 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:23.305631 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:23.305632 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:23.305634 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:33:23.305717 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:23.305720 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:23.306383 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:23.306384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:23.307705 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:23.307706 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:23.308754 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:23.308755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:23.308847 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:23.308851 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:33:23.309165 (+ 314us) client_negotiation.cc:240] Negotiation successful
0928 06:33:23.309209 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":170,"thread_start_us":93,"threads_started":1}
I20260928 06:33:23.309549 18565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e3e4c32f4cc472a9f838fcb8d0e3059" candidate_uuid: "02a32df533b64624934cc9f57e59fc8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ca55a791204376a9ddd1bd72b4cff6" is_pre_election: true
I20260928 06:33:23.309669 18285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e3e4c32f4cc472a9f838fcb8d0e3059" candidate_uuid: "02a32df533b64624934cc9f57e59fc8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cf20e908f614fea9452dbe848d1dac3" is_pre_election: true
I20260928 06:33:23.309795 18565 raft_consensus.cc:2454] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P a5ca55a791204376a9ddd1bd72b4cff6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02a32df533b64624934cc9f57e59fc8b in term 0.
I20260928 06:33:23.309837 18285 raft_consensus.cc:2454] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P 8cf20e908f614fea9452dbe848d1dac3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02a32df533b64624934cc9f57e59fc8b in term 0.
I20260928 06:33:23.310070 18358 leader_election.cc:302] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P 02a32df533b64624934cc9f57e59fc8b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02a32df533b64624934cc9f57e59fc8b, 8cf20e908f614fea9452dbe848d1dac3; no voters:
I20260928 06:33:23.310227 18642 raft_consensus.cc:2781] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P 02a32df533b64624934cc9f57e59fc8b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:33:23.310286 18642 raft_consensus.cc:482] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P 02a32df533b64624934cc9f57e59fc8b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:33:23.310312 18642 raft_consensus.cc:3037] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P 02a32df533b64624934cc9f57e59fc8b [term 0 FOLLOWER]: Advancing to term 1
W20260928 06:33:23.310511 18612 tablet.cc:2369] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P a5ca55a791204376a9ddd1bd72b4cff6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:33:23.311379 18642 raft_consensus.cc:504] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P 02a32df533b64624934cc9f57e59fc8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cf20e908f614fea9452dbe848d1dac3" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 35783 } } peers { permanent_uuid: "02a32df533b64624934cc9f57e59fc8b" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 33503 } } peers { permanent_uuid: "a5ca55a791204376a9ddd1bd72b4cff6" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39679 } }
I20260928 06:33:23.311632 18642 leader_election.cc:288] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P 02a32df533b64624934cc9f57e59fc8b [CANDIDATE]: Term 1 election: Requested vote from peers 8cf20e908f614fea9452dbe848d1dac3 (127.16.157.65:35783), a5ca55a791204376a9ddd1bd72b4cff6 (127.16.157.67:39679)
I20260928 06:33:23.311679 18285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e3e4c32f4cc472a9f838fcb8d0e3059" candidate_uuid: "02a32df533b64624934cc9f57e59fc8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cf20e908f614fea9452dbe848d1dac3"
I20260928 06:33:23.311760 18285 raft_consensus.cc:3037] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P 8cf20e908f614fea9452dbe848d1dac3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:23.311784 18565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e3e4c32f4cc472a9f838fcb8d0e3059" candidate_uuid: "02a32df533b64624934cc9f57e59fc8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ca55a791204376a9ddd1bd72b4cff6"
I20260928 06:33:23.311853 18565 raft_consensus.cc:3037] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P a5ca55a791204376a9ddd1bd72b4cff6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:23.312556 18565 raft_consensus.cc:2454] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P a5ca55a791204376a9ddd1bd72b4cff6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02a32df533b64624934cc9f57e59fc8b in term 1.
I20260928 06:33:23.312739 18357 leader_election.cc:302] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P 02a32df533b64624934cc9f57e59fc8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02a32df533b64624934cc9f57e59fc8b, a5ca55a791204376a9ddd1bd72b4cff6; no voters:
I20260928 06:33:23.312829 18642 raft_consensus.cc:2781] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P 02a32df533b64624934cc9f57e59fc8b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:33:23.312978 18285 raft_consensus.cc:2454] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P 8cf20e908f614fea9452dbe848d1dac3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02a32df533b64624934cc9f57e59fc8b in term 1.
I20260928 06:33:23.312990 18642 raft_consensus.cc:686] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P 02a32df533b64624934cc9f57e59fc8b [term 1 LEADER]: Becoming Leader. State: Replica: 02a32df533b64624934cc9f57e59fc8b, State: Running, Role: LEADER
I20260928 06:33:23.313095 18642 consensus_queue.cc:237] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P 02a32df533b64624934cc9f57e59fc8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cf20e908f614fea9452dbe848d1dac3" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 35783 } } peers { permanent_uuid: "02a32df533b64624934cc9f57e59fc8b" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 33503 } } peers { permanent_uuid: "a5ca55a791204376a9ddd1bd72b4cff6" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39679 } }
I20260928 06:33:23.314095 18126 catalog_manager.cc:5495] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P 02a32df533b64624934cc9f57e59fc8b reported cstate change: term changed from 0 to 1, leader changed from <none> to 02a32df533b64624934cc9f57e59fc8b (127.16.157.66). New cstate: current_term: 1 leader_uuid: "02a32df533b64624934cc9f57e59fc8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cf20e908f614fea9452dbe848d1dac3" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 35783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02a32df533b64624934cc9f57e59fc8b" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 33503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5ca55a791204376a9ddd1bd72b4cff6" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39679 } health_report { overall_health: UNKNOWN } } }
I20260928 06:33:23.337077 18632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:23.333932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54420 (local address 127.16.157.66:33503)
0928 06:33:23.333976 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:23.333978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:23.334018 (+ 40us) server_negotiation.cc:407] Connection header received
0928 06:33:23.334153 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:23.334153 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:23.334171 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:23.334240 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:33:23.334436 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:23.334898 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:23.336104 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:23.336285 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:23.336332 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:23.336635 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:33:23.336775 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:33:23.336837 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:23.336947 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 06:33:23.336966 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:33:23.339200 18285 raft_consensus.cc:1261] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P 8cf20e908f614fea9452dbe848d1dac3 [term 1 FOLLOWER]: Refusing update from remote peer 02a32df533b64624934cc9f57e59fc8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:33:23.339227 18565 raft_consensus.cc:1261] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P a5ca55a791204376a9ddd1bd72b4cff6 [term 1 FOLLOWER]: Refusing update from remote peer 02a32df533b64624934cc9f57e59fc8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:33:23.339533 18642 consensus_queue.cc:1035] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P 02a32df533b64624934cc9f57e59fc8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cf20e908f614fea9452dbe848d1dac3" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 35783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:33:23.339671 18644 consensus_queue.cc:1035] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P 02a32df533b64624934cc9f57e59fc8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca55a791204376a9ddd1bd72b4cff6" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:33:23.343371 18648 mvcc.cc:205] Tried to move back new op lower bound from 7334204224874078208 to 7334204224771268608. Current Snapshot: MvccSnapshot[applied={T|T < 7334204224874078208}]
I20260928 06:33:23.346010 18126 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:33:23.346109 18126 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:33:23.346915 18126 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ee1eda9c7d684f88a77cb007b5502e30: Sending DeleteTablet for 3 replicas of tablet 4e3e4c32f4cc472a9f838fcb8d0e3059
I20260928 06:33:23.347302 18545 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e3e4c32f4cc472a9f838fcb8d0e3059 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:33:23 UTC) from {username='kudu', principal='kudu/127.16.157.126@KRBTEST.COM'} at 127.0.0.1:36364
I20260928 06:33:23.347318 18265 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e3e4c32f4cc472a9f838fcb8d0e3059 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:33:23 UTC) from {username='kudu', principal='kudu/127.16.157.126@KRBTEST.COM'} at 127.0.0.1:41740
I20260928 06:33:23.347600 18405 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e3e4c32f4cc472a9f838fcb8d0e3059 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:33:23 UTC) from {username='kudu', principal='kudu/127.16.157.126@KRBTEST.COM'} at 127.0.0.1:54412
I20260928 06:33:23.347836 18659 tablet_replica.cc:330] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P 02a32df533b64624934cc9f57e59fc8b: stopping tablet replica
I20260928 06:33:23.347994 18659 raft_consensus.cc:2229] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P 02a32df533b64624934cc9f57e59fc8b [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:33:23.348097 18126 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33252:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:33:23.348261 18126 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:33:23.348208 18659 raft_consensus.cc:2258] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P 02a32df533b64624934cc9f57e59fc8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:33:23.348958 18659 ts_tablet_manager.cc:1905] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P 02a32df533b64624934cc9f57e59fc8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:33:23.349648 18265 tablet_service.cc:1467] Processing CreateTablet for tablet 7df6f425df9a44c4bcf27c37e13d708b (DEFAULT_TABLE table=test-table [id=ce1761ade9004ad9ba8a496b998d1ed2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:33:23.349792 18265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7df6f425df9a44c4bcf27c37e13d708b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:23.351163 18636 tablet_bootstrap.cc:492] T 7df6f425df9a44c4bcf27c37e13d708b P 8cf20e908f614fea9452dbe848d1dac3: Bootstrap starting.
I20260928 06:33:23.351287 18659 ts_tablet_manager.cc:1918] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P 02a32df533b64624934cc9f57e59fc8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:33:23.351380 18405 tablet_service.cc:1467] Processing CreateTablet for tablet 7df6f425df9a44c4bcf27c37e13d708b (DEFAULT_TABLE table=test-table [id=ce1761ade9004ad9ba8a496b998d1ed2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:33:23.351413 18659 log.cc:1196] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P 02a32df533b64624934cc9f57e59fc8b: Deleting WAL directory at /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/ts-1/wal/wals/4e3e4c32f4cc472a9f838fcb8d0e3059
I20260928 06:33:23.351573 18636 tablet_bootstrap.cc:654] T 7df6f425df9a44c4bcf27c37e13d708b P 8cf20e908f614fea9452dbe848d1dac3: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:23.351745 18545 tablet_service.cc:1467] Processing CreateTablet for tablet 7df6f425df9a44c4bcf27c37e13d708b (DEFAULT_TABLE table=test-table [id=ce1761ade9004ad9ba8a496b998d1ed2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:33:23.351876 18545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7df6f425df9a44c4bcf27c37e13d708b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:23.352007 18659 ts_tablet_manager.cc:1939] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P 02a32df533b64624934cc9f57e59fc8b: Deleting consensus metadata
I20260928 06:33:23.353650 18634 tablet_bootstrap.cc:492] T 7df6f425df9a44c4bcf27c37e13d708b P a5ca55a791204376a9ddd1bd72b4cff6: Bootstrap starting.
I20260928 06:33:23.353772 18114 catalog_manager.cc:4841] TS 02a32df533b64624934cc9f57e59fc8b (127.16.157.66:33503): tablet 4e3e4c32f4cc472a9f838fcb8d0e3059 (table test-table [id=c661596f0a674ede9551b094579e7502]) successfully deleted
I20260928 06:33:23.351493 18405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7df6f425df9a44c4bcf27c37e13d708b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:23.354132 18634 tablet_bootstrap.cc:654] T 7df6f425df9a44c4bcf27c37e13d708b P a5ca55a791204376a9ddd1bd72b4cff6: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:23.355175 18639 tablet_bootstrap.cc:492] T 7df6f425df9a44c4bcf27c37e13d708b P 02a32df533b64624934cc9f57e59fc8b: Bootstrap starting.
I20260928 06:33:23.355624 18639 tablet_bootstrap.cc:654] T 7df6f425df9a44c4bcf27c37e13d708b P 02a32df533b64624934cc9f57e59fc8b: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:23.355806 18657 tablet_replica.cc:330] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P a5ca55a791204376a9ddd1bd72b4cff6: stopping tablet replica
I20260928 06:33:23.355926 18657 raft_consensus.cc:2229] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P a5ca55a791204376a9ddd1bd72b4cff6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:33:23.356065 18657 raft_consensus.cc:2258] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P a5ca55a791204376a9ddd1bd72b4cff6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:33:23.356353 18658 tablet_replica.cc:330] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P 8cf20e908f614fea9452dbe848d1dac3: stopping tablet replica
I20260928 06:33:23.356474 18658 raft_consensus.cc:2229] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P 8cf20e908f614fea9452dbe848d1dac3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:33:23.356585 18658 raft_consensus.cc:2258] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P 8cf20e908f614fea9452dbe848d1dac3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:33:23.356910 18657 ts_tablet_manager.cc:1905] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P a5ca55a791204376a9ddd1bd72b4cff6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:33:23.359365 18657 ts_tablet_manager.cc:1918] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P a5ca55a791204376a9ddd1bd72b4cff6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:33:23.359442 18657 log.cc:1196] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P a5ca55a791204376a9ddd1bd72b4cff6: Deleting WAL directory at /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/ts-2/wal/wals/4e3e4c32f4cc472a9f838fcb8d0e3059
I20260928 06:33:23.359705 18657 ts_tablet_manager.cc:1939] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P a5ca55a791204376a9ddd1bd72b4cff6: Deleting consensus metadata
I20260928 06:33:23.360109 18113 catalog_manager.cc:4841] TS a5ca55a791204376a9ddd1bd72b4cff6 (127.16.157.67:39679): tablet 4e3e4c32f4cc472a9f838fcb8d0e3059 (table test-table [id=c661596f0a674ede9551b094579e7502]) successfully deleted
I20260928 06:33:23.360466 18658 ts_tablet_manager.cc:1905] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P 8cf20e908f614fea9452dbe848d1dac3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:33:23.362318 18658 ts_tablet_manager.cc:1918] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P 8cf20e908f614fea9452dbe848d1dac3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:33:23.362394 18658 log.cc:1196] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P 8cf20e908f614fea9452dbe848d1dac3: Deleting WAL directory at /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/ts-0/wal/wals/4e3e4c32f4cc472a9f838fcb8d0e3059
I20260928 06:33:23.362646 18658 ts_tablet_manager.cc:1939] T 4e3e4c32f4cc472a9f838fcb8d0e3059 P 8cf20e908f614fea9452dbe848d1dac3: Deleting consensus metadata
I20260928 06:33:23.363174 18114 catalog_manager.cc:4841] TS 8cf20e908f614fea9452dbe848d1dac3 (127.16.157.65:35783): tablet 4e3e4c32f4cc472a9f838fcb8d0e3059 (table test-table [id=c661596f0a674ede9551b094579e7502]) successfully deleted
I20260928 06:33:23.366086 18636 tablet_bootstrap.cc:492] T 7df6f425df9a44c4bcf27c37e13d708b P 8cf20e908f614fea9452dbe848d1dac3: No bootstrap required, opened a new log
I20260928 06:33:23.366158 18636 ts_tablet_manager.cc:1397] T 7df6f425df9a44c4bcf27c37e13d708b P 8cf20e908f614fea9452dbe848d1dac3: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.000s
I20260928 06:33:23.366542 18636 raft_consensus.cc:348] T 7df6f425df9a44c4bcf27c37e13d708b P 8cf20e908f614fea9452dbe848d1dac3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cf20e908f614fea9452dbe848d1dac3" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 35783 } } peers { permanent_uuid: "02a32df533b64624934cc9f57e59fc8b" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 33503 } } peers { permanent_uuid: "a5ca55a791204376a9ddd1bd72b4cff6" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39679 } }
I20260928 06:33:23.366643 18636 raft_consensus.cc:374] T 7df6f425df9a44c4bcf27c37e13d708b P 8cf20e908f614fea9452dbe848d1dac3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:23.366667 18636 raft_consensus.cc:729] T 7df6f425df9a44c4bcf27c37e13d708b P 8cf20e908f614fea9452dbe848d1dac3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cf20e908f614fea9452dbe848d1dac3, State: Initialized, Role: FOLLOWER
I20260928 06:33:23.366724 18636 consensus_queue.cc:260] T 7df6f425df9a44c4bcf27c37e13d708b P 8cf20e908f614fea9452dbe848d1dac3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cf20e908f614fea9452dbe848d1dac3" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 35783 } } peers { permanent_uuid: "02a32df533b64624934cc9f57e59fc8b" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 33503 } } peers { permanent_uuid: "a5ca55a791204376a9ddd1bd72b4cff6" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39679 } }
I20260928 06:33:23.366842 18636 ts_tablet_manager.cc:1428] T 7df6f425df9a44c4bcf27c37e13d708b P 8cf20e908f614fea9452dbe848d1dac3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:23.367257 18639 tablet_bootstrap.cc:492] T 7df6f425df9a44c4bcf27c37e13d708b P 02a32df533b64624934cc9f57e59fc8b: No bootstrap required, opened a new log
I20260928 06:33:23.367324 18639 ts_tablet_manager.cc:1397] T 7df6f425df9a44c4bcf27c37e13d708b P 02a32df533b64624934cc9f57e59fc8b: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.000s
I20260928 06:33:23.367492 18639 raft_consensus.cc:348] T 7df6f425df9a44c4bcf27c37e13d708b P 02a32df533b64624934cc9f57e59fc8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cf20e908f614fea9452dbe848d1dac3" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 35783 } } peers { permanent_uuid: "02a32df533b64624934cc9f57e59fc8b" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 33503 } } peers { permanent_uuid: "a5ca55a791204376a9ddd1bd72b4cff6" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39679 } }
I20260928 06:33:23.367594 18639 raft_consensus.cc:374] T 7df6f425df9a44c4bcf27c37e13d708b P 02a32df533b64624934cc9f57e59fc8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:23.367617 18639 raft_consensus.cc:729] T 7df6f425df9a44c4bcf27c37e13d708b P 02a32df533b64624934cc9f57e59fc8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02a32df533b64624934cc9f57e59fc8b, State: Initialized, Role: FOLLOWER
I20260928 06:33:23.367664 18639 consensus_queue.cc:260] T 7df6f425df9a44c4bcf27c37e13d708b P 02a32df533b64624934cc9f57e59fc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cf20e908f614fea9452dbe848d1dac3" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 35783 } } peers { permanent_uuid: "02a32df533b64624934cc9f57e59fc8b" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 33503 } } peers { permanent_uuid: "a5ca55a791204376a9ddd1bd72b4cff6" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39679 } }
I20260928 06:33:23.367765 18639 ts_tablet_manager.cc:1428] T 7df6f425df9a44c4bcf27c37e13d708b P 02a32df533b64624934cc9f57e59fc8b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 06:33:23.368211 18634 tablet_bootstrap.cc:492] T 7df6f425df9a44c4bcf27c37e13d708b P a5ca55a791204376a9ddd1bd72b4cff6: No bootstrap required, opened a new log
I20260928 06:33:23.368275 18634 ts_tablet_manager.cc:1397] T 7df6f425df9a44c4bcf27c37e13d708b P a5ca55a791204376a9ddd1bd72b4cff6: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.000s
I20260928 06:33:23.368460 18634 raft_consensus.cc:348] T 7df6f425df9a44c4bcf27c37e13d708b P a5ca55a791204376a9ddd1bd72b4cff6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cf20e908f614fea9452dbe848d1dac3" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 35783 } } peers { permanent_uuid: "02a32df533b64624934cc9f57e59fc8b" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 33503 } } peers { permanent_uuid: "a5ca55a791204376a9ddd1bd72b4cff6" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39679 } }
I20260928 06:33:23.368676 18634 raft_consensus.cc:374] T 7df6f425df9a44c4bcf27c37e13d708b P a5ca55a791204376a9ddd1bd72b4cff6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:23.368726 18634 raft_consensus.cc:729] T 7df6f425df9a44c4bcf27c37e13d708b P a5ca55a791204376a9ddd1bd72b4cff6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5ca55a791204376a9ddd1bd72b4cff6, State: Initialized, Role: FOLLOWER
I20260928 06:33:23.368775 18634 consensus_queue.cc:260] T 7df6f425df9a44c4bcf27c37e13d708b P a5ca55a791204376a9ddd1bd72b4cff6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cf20e908f614fea9452dbe848d1dac3" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 35783 } } peers { permanent_uuid: "02a32df533b64624934cc9f57e59fc8b" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 33503 } } peers { permanent_uuid: "a5ca55a791204376a9ddd1bd72b4cff6" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39679 } }
I20260928 06:33:23.368875 18634 ts_tablet_manager.cc:1428] T 7df6f425df9a44c4bcf27c37e13d708b P a5ca55a791204376a9ddd1bd72b4cff6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20260928 06:33:23.372128 18472 tablet.cc:2369] T 7df6f425df9a44c4bcf27c37e13d708b P 02a32df533b64624934cc9f57e59fc8b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:33:23.458982 18637 raft_consensus.cc:482] T 7df6f425df9a44c4bcf27c37e13d708b P a5ca55a791204376a9ddd1bd72b4cff6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:33:23.459105 18637 raft_consensus.cc:504] T 7df6f425df9a44c4bcf27c37e13d708b P a5ca55a791204376a9ddd1bd72b4cff6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cf20e908f614fea9452dbe848d1dac3" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 35783 } } peers { permanent_uuid: "02a32df533b64624934cc9f57e59fc8b" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 33503 } } peers { permanent_uuid: "a5ca55a791204376a9ddd1bd72b4cff6" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39679 } }
I20260928 06:33:23.459472 18637 leader_election.cc:288] T 7df6f425df9a44c4bcf27c37e13d708b P a5ca55a791204376a9ddd1bd72b4cff6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8cf20e908f614fea9452dbe848d1dac3 (127.16.157.65:35783), 02a32df533b64624934cc9f57e59fc8b (127.16.157.66:33503)
I20260928 06:33:23.463650 18631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:23.459967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:53271 (local address 127.16.157.65:35783)
0928 06:33:23.460012 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:23.460014 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:23.460022 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:33:23.460051 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:23.460052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:23.460061 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:23.460112 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:33:23.460332 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:23.460997 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:23.462569 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:23.463121 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:23.463201 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:23.463211 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:23.463486 (+ 275us) server_negotiation.cc:299] Negotiation successful
0928 06:33:23.463533 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 06:33:23.463712 18632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:23.459949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:33079 (local address 127.16.157.66:33503)
0928 06:33:23.459994 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:23.459996 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:23.460206 (+ 210us) server_negotiation.cc:407] Connection header received
0928 06:33:23.460305 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:23.460306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:23.460316 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:23.460344 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:33:23.460562 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:23.461261 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:23.462788 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:23.463305 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:23.463374 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:23.463383 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:23.463599 (+ 216us) server_negotiation.cc:299] Negotiation successful
0928 06:33:23.463639 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:33:23.463940 18504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:23.459706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.65:35783 (local address 127.16.157.67:53271)
0928 06:33:23.459793 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:23.459807 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:23.459873 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:23.460164 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:23.460165 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:23.460167 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:33:23.460239 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:23.460241 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:23.461102 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:23.461103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:23.462410 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:23.462413 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:23.463244 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:23.463245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:23.463338 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:23.463342 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:33:23.463395 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 06:33:23.463853 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260928 06:33:23.464190 18285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7df6f425df9a44c4bcf27c37e13d708b" candidate_uuid: "a5ca55a791204376a9ddd1bd72b4cff6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cf20e908f614fea9452dbe848d1dac3" is_pre_election: true
I20260928 06:33:23.464314 18285 raft_consensus.cc:2454] T 7df6f425df9a44c4bcf27c37e13d708b P 8cf20e908f614fea9452dbe848d1dac3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5ca55a791204376a9ddd1bd72b4cff6 in term 0.
I20260928 06:33:23.464435 18663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:23.459803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.66:33503 (local address 127.16.157.67:33079)
0928 06:33:23.460128 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:23.460140 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:23.460233 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:23.460405 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:23.460406 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:23.460408 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:33:23.460493 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:23.460495 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:23.461357 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:23.461358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:23.462676 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:23.462680 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:23.463454 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:23.463455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:23.463523 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:23.463526 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:33:23.464204 (+ 678us) client_negotiation.cc:240] Negotiation successful
0928 06:33:23.464255 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":61,"threads_started":1}
I20260928 06:33:23.464504 18498 leader_election.cc:302] T 7df6f425df9a44c4bcf27c37e13d708b P a5ca55a791204376a9ddd1bd72b4cff6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8cf20e908f614fea9452dbe848d1dac3, a5ca55a791204376a9ddd1bd72b4cff6; no voters:
I20260928 06:33:23.464740 18637 raft_consensus.cc:2781] T 7df6f425df9a44c4bcf27c37e13d708b P a5ca55a791204376a9ddd1bd72b4cff6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:33:23.464783 18637 raft_consensus.cc:482] T 7df6f425df9a44c4bcf27c37e13d708b P a5ca55a791204376a9ddd1bd72b4cff6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:33:23.464803 18637 raft_consensus.cc:3037] T 7df6f425df9a44c4bcf27c37e13d708b P a5ca55a791204376a9ddd1bd72b4cff6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:23.464871 18425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7df6f425df9a44c4bcf27c37e13d708b" candidate_uuid: "a5ca55a791204376a9ddd1bd72b4cff6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02a32df533b64624934cc9f57e59fc8b" is_pre_election: true
I20260928 06:33:23.465021 18425 raft_consensus.cc:2454] T 7df6f425df9a44c4bcf27c37e13d708b P 02a32df533b64624934cc9f57e59fc8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5ca55a791204376a9ddd1bd72b4cff6 in term 0.
I20260928 06:33:23.465832 18637 raft_consensus.cc:504] T 7df6f425df9a44c4bcf27c37e13d708b P a5ca55a791204376a9ddd1bd72b4cff6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cf20e908f614fea9452dbe848d1dac3" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 35783 } } peers { permanent_uuid: "02a32df533b64624934cc9f57e59fc8b" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 33503 } } peers { permanent_uuid: "a5ca55a791204376a9ddd1bd72b4cff6" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39679 } }
I20260928 06:33:23.465986 18637 leader_election.cc:288] T 7df6f425df9a44c4bcf27c37e13d708b P a5ca55a791204376a9ddd1bd72b4cff6 [CANDIDATE]: Term 1 election: Requested vote from peers 8cf20e908f614fea9452dbe848d1dac3 (127.16.157.65:35783), 02a32df533b64624934cc9f57e59fc8b (127.16.157.66:33503)
I20260928 06:33:23.466245 18285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7df6f425df9a44c4bcf27c37e13d708b" candidate_uuid: "a5ca55a791204376a9ddd1bd72b4cff6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cf20e908f614fea9452dbe848d1dac3"
I20260928 06:33:23.466264 18425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7df6f425df9a44c4bcf27c37e13d708b" candidate_uuid: "a5ca55a791204376a9ddd1bd72b4cff6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02a32df533b64624934cc9f57e59fc8b"
I20260928 06:33:23.466331 18285 raft_consensus.cc:3037] T 7df6f425df9a44c4bcf27c37e13d708b P 8cf20e908f614fea9452dbe848d1dac3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:23.466351 18425 raft_consensus.cc:3037] T 7df6f425df9a44c4bcf27c37e13d708b P 02a32df533b64624934cc9f57e59fc8b [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:23.467097 18285 raft_consensus.cc:2454] T 7df6f425df9a44c4bcf27c37e13d708b P 8cf20e908f614fea9452dbe848d1dac3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5ca55a791204376a9ddd1bd72b4cff6 in term 1.
I20260928 06:33:23.467193 18425 raft_consensus.cc:2454] T 7df6f425df9a44c4bcf27c37e13d708b P 02a32df533b64624934cc9f57e59fc8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5ca55a791204376a9ddd1bd72b4cff6 in term 1.
I20260928 06:33:23.467303 18498 leader_election.cc:302] T 7df6f425df9a44c4bcf27c37e13d708b P a5ca55a791204376a9ddd1bd72b4cff6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8cf20e908f614fea9452dbe848d1dac3, a5ca55a791204376a9ddd1bd72b4cff6; no voters:
I20260928 06:33:23.467409 18637 raft_consensus.cc:2781] T 7df6f425df9a44c4bcf27c37e13d708b P a5ca55a791204376a9ddd1bd72b4cff6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:33:23.467527 18637 raft_consensus.cc:686] T 7df6f425df9a44c4bcf27c37e13d708b P a5ca55a791204376a9ddd1bd72b4cff6 [term 1 LEADER]: Becoming Leader. State: Replica: a5ca55a791204376a9ddd1bd72b4cff6, State: Running, Role: LEADER
I20260928 06:33:23.467592 18637 consensus_queue.cc:237] T 7df6f425df9a44c4bcf27c37e13d708b P a5ca55a791204376a9ddd1bd72b4cff6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cf20e908f614fea9452dbe848d1dac3" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 35783 } } peers { permanent_uuid: "02a32df533b64624934cc9f57e59fc8b" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 33503 } } peers { permanent_uuid: "a5ca55a791204376a9ddd1bd72b4cff6" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39679 } }
I20260928 06:33:23.468402 18126 catalog_manager.cc:5495] T 7df6f425df9a44c4bcf27c37e13d708b P a5ca55a791204376a9ddd1bd72b4cff6 reported cstate change: term changed from 0 to 1, leader changed from <none> to a5ca55a791204376a9ddd1bd72b4cff6 (127.16.157.67). New cstate: current_term: 1 leader_uuid: "a5ca55a791204376a9ddd1bd72b4cff6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cf20e908f614fea9452dbe848d1dac3" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 35783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02a32df533b64624934cc9f57e59fc8b" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 33503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5ca55a791204376a9ddd1bd72b4cff6" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39679 } health_report { overall_health: HEALTHY } } }
Sep 28 06:33:23 dist-test-slave-rrj5 krb5kdc[18083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577202, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.126@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:33:23.505985 18671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:23.498427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40707 (local address 127.0.0.1:33264)
0928 06:33:23.498621 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:23.498632 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:23.498735 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:23.498908 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:23.498909 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:23.499226 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:23.499281 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:23.499284 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:23.500238 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:23.500239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:23.500799 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:23.500801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:23.501265 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:23.501266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:23.501308 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:23.501925 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:23.501929 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:23.503262 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:23.504960 (+ 1698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:23.504968 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:23.504970 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:23.505165 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:23.505367 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:23.505368 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:23.505368 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:23.505424 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:23.505704 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:23.505710 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:23.505753 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 06:33:23.505819 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 06:33:23.505910 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":125,"threads_started":1}
I20260928 06:33:23.506300 18178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:23.498512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33264 (local address 127.16.157.126:40707)
0928 06:33:23.498587 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:23.498589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:23.498713 (+ 124us) server_negotiation.cc:407] Connection header received
0928 06:33:23.498799 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:23.498800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:23.498812 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:23.498869 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:23.499365 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:23.500144 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:23.500878 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:23.501189 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:23.501258 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:23.503353 (+ 2095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:23.503355 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:23.503355 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:23.503362 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:23.504878 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:23.505236 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:23.505237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:23.505238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:23.505300 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:23.505484 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:23.505485 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:23.505485 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:23.505638 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:23.505708 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:23.505866 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 06:33:23.505975 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":41}
I20260928 06:33:23.508379 18126 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.16.157.126@KRBTEST.COM'} at 127.0.0.1:33264:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 06:33:23.508651 18126 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:33:23.510192 18545 tablet_service.cc:1467] Processing CreateTablet for tablet e9596bb0ecd24bfcb9f91493eb23dfa1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=51935175d9d245a2b2297499c579ac7a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:33:23.510246 18405 tablet_service.cc:1467] Processing CreateTablet for tablet e9596bb0ecd24bfcb9f91493eb23dfa1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=51935175d9d245a2b2297499c579ac7a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:33:23.510231 18265 tablet_service.cc:1467] Processing CreateTablet for tablet e9596bb0ecd24bfcb9f91493eb23dfa1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=51935175d9d245a2b2297499c579ac7a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:33:23.510366 18405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9596bb0ecd24bfcb9f91493eb23dfa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:23.510365 18545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9596bb0ecd24bfcb9f91493eb23dfa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:23.510393 18265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9596bb0ecd24bfcb9f91493eb23dfa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:23.511938 18639 tablet_bootstrap.cc:492] T e9596bb0ecd24bfcb9f91493eb23dfa1 P 02a32df533b64624934cc9f57e59fc8b: Bootstrap starting.
I20260928 06:33:23.511974 18634 tablet_bootstrap.cc:492] T e9596bb0ecd24bfcb9f91493eb23dfa1 P a5ca55a791204376a9ddd1bd72b4cff6: Bootstrap starting.
I20260928 06:33:23.512225 18636 tablet_bootstrap.cc:492] T e9596bb0ecd24bfcb9f91493eb23dfa1 P 8cf20e908f614fea9452dbe848d1dac3: Bootstrap starting.
I20260928 06:33:23.512310 18639 tablet_bootstrap.cc:654] T e9596bb0ecd24bfcb9f91493eb23dfa1 P 02a32df533b64624934cc9f57e59fc8b: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:23.512365 18634 tablet_bootstrap.cc:654] T e9596bb0ecd24bfcb9f91493eb23dfa1 P a5ca55a791204376a9ddd1bd72b4cff6: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:23.512689 18636 tablet_bootstrap.cc:654] T e9596bb0ecd24bfcb9f91493eb23dfa1 P 8cf20e908f614fea9452dbe848d1dac3: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:23.513129 18634 tablet_bootstrap.cc:492] T e9596bb0ecd24bfcb9f91493eb23dfa1 P a5ca55a791204376a9ddd1bd72b4cff6: No bootstrap required, opened a new log
I20260928 06:33:23.513200 18634 ts_tablet_manager.cc:1397] T e9596bb0ecd24bfcb9f91493eb23dfa1 P a5ca55a791204376a9ddd1bd72b4cff6: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260928 06:33:23.513262 18639 tablet_bootstrap.cc:492] T e9596bb0ecd24bfcb9f91493eb23dfa1 P 02a32df533b64624934cc9f57e59fc8b: No bootstrap required, opened a new log
I20260928 06:33:23.513314 18639 ts_tablet_manager.cc:1397] T e9596bb0ecd24bfcb9f91493eb23dfa1 P 02a32df533b64624934cc9f57e59fc8b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:23.513357 18634 raft_consensus.cc:348] T e9596bb0ecd24bfcb9f91493eb23dfa1 P a5ca55a791204376a9ddd1bd72b4cff6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cf20e908f614fea9452dbe848d1dac3" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 35783 } } peers { permanent_uuid: "02a32df533b64624934cc9f57e59fc8b" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 33503 } } peers { permanent_uuid: "a5ca55a791204376a9ddd1bd72b4cff6" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39679 } }
I20260928 06:33:23.513437 18634 raft_consensus.cc:374] T e9596bb0ecd24bfcb9f91493eb23dfa1 P a5ca55a791204376a9ddd1bd72b4cff6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:23.513458 18634 raft_consensus.cc:729] T e9596bb0ecd24bfcb9f91493eb23dfa1 P a5ca55a791204376a9ddd1bd72b4cff6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5ca55a791204376a9ddd1bd72b4cff6, State: Initialized, Role: FOLLOWER
I20260928 06:33:23.513423 18639 raft_consensus.cc:348] T e9596bb0ecd24bfcb9f91493eb23dfa1 P 02a32df533b64624934cc9f57e59fc8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cf20e908f614fea9452dbe848d1dac3" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 35783 } } peers { permanent_uuid: "02a32df533b64624934cc9f57e59fc8b" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 33503 } } peers { permanent_uuid: "a5ca55a791204376a9ddd1bd72b4cff6" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39679 } }
I20260928 06:33:23.513492 18639 raft_consensus.cc:374] T e9596bb0ecd24bfcb9f91493eb23dfa1 P 02a32df533b64624934cc9f57e59fc8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:23.513520 18639 raft_consensus.cc:729] T e9596bb0ecd24bfcb9f91493eb23dfa1 P 02a32df533b64624934cc9f57e59fc8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02a32df533b64624934cc9f57e59fc8b, State: Initialized, Role: FOLLOWER
I20260928 06:33:23.513509 18634 consensus_queue.cc:260] T e9596bb0ecd24bfcb9f91493eb23dfa1 P a5ca55a791204376a9ddd1bd72b4cff6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cf20e908f614fea9452dbe848d1dac3" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 35783 } } peers { permanent_uuid: "02a32df533b64624934cc9f57e59fc8b" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 33503 } } peers { permanent_uuid: "a5ca55a791204376a9ddd1bd72b4cff6" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39679 } }
I20260928 06:33:23.513589 18639 consensus_queue.cc:260] T e9596bb0ecd24bfcb9f91493eb23dfa1 P 02a32df533b64624934cc9f57e59fc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cf20e908f614fea9452dbe848d1dac3" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 35783 } } peers { permanent_uuid: "02a32df533b64624934cc9f57e59fc8b" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 33503 } } peers { permanent_uuid: "a5ca55a791204376a9ddd1bd72b4cff6" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39679 } }
I20260928 06:33:23.513648 18634 ts_tablet_manager.cc:1428] T e9596bb0ecd24bfcb9f91493eb23dfa1 P a5ca55a791204376a9ddd1bd72b4cff6: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 06:33:23.513695 18639 ts_tablet_manager.cc:1428] T e9596bb0ecd24bfcb9f91493eb23dfa1 P 02a32df533b64624934cc9f57e59fc8b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:23.513756 18637 tablet_replica.cc:439] T e9596bb0ecd24bfcb9f91493eb23dfa1 P a5ca55a791204376a9ddd1bd72b4cff6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cf20e908f614fea9452dbe848d1dac3" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 35783 } } peers { permanent_uuid: "02a32df533b64624934cc9f57e59fc8b" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 33503 } } peers { permanent_uuid: "a5ca55a791204376a9ddd1bd72b4cff6" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39679 } } }
I20260928 06:33:23.513875 18637 tablet_replica.cc:442] T e9596bb0ecd24bfcb9f91493eb23dfa1 P a5ca55a791204376a9ddd1bd72b4cff6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:33:23.513893 18644 tablet_replica.cc:439] T e9596bb0ecd24bfcb9f91493eb23dfa1 P 02a32df533b64624934cc9f57e59fc8b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cf20e908f614fea9452dbe848d1dac3" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 35783 } } peers { permanent_uuid: "02a32df533b64624934cc9f57e59fc8b" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 33503 } } peers { permanent_uuid: "a5ca55a791204376a9ddd1bd72b4cff6" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39679 } } }
I20260928 06:33:23.513994 18644 tablet_replica.cc:442] T e9596bb0ecd24bfcb9f91493eb23dfa1 P 02a32df533b64624934cc9f57e59fc8b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:33:23.514262 18636 tablet_bootstrap.cc:492] T e9596bb0ecd24bfcb9f91493eb23dfa1 P 8cf20e908f614fea9452dbe848d1dac3: No bootstrap required, opened a new log
I20260928 06:33:23.514325 18636 ts_tablet_manager.cc:1397] T e9596bb0ecd24bfcb9f91493eb23dfa1 P 8cf20e908f614fea9452dbe848d1dac3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:33:23.514472 18636 raft_consensus.cc:348] T e9596bb0ecd24bfcb9f91493eb23dfa1 P 8cf20e908f614fea9452dbe848d1dac3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cf20e908f614fea9452dbe848d1dac3" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 35783 } } peers { permanent_uuid: "02a32df533b64624934cc9f57e59fc8b" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 33503 } } peers { permanent_uuid: "a5ca55a791204376a9ddd1bd72b4cff6" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39679 } }
I20260928 06:33:23.514545 18636 raft_consensus.cc:374] T e9596bb0ecd24bfcb9f91493eb23dfa1 P 8cf20e908f614fea9452dbe848d1dac3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:23.514568 18636 raft_consensus.cc:729] T e9596bb0ecd24bfcb9f91493eb23dfa1 P 8cf20e908f614fea9452dbe848d1dac3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cf20e908f614fea9452dbe848d1dac3, State: Initialized, Role: FOLLOWER
I20260928 06:33:23.514611 18636 consensus_queue.cc:260] T e9596bb0ecd24bfcb9f91493eb23dfa1 P 8cf20e908f614fea9452dbe848d1dac3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cf20e908f614fea9452dbe848d1dac3" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 35783 } } peers { permanent_uuid: "02a32df533b64624934cc9f57e59fc8b" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 33503 } } peers { permanent_uuid: "a5ca55a791204376a9ddd1bd72b4cff6" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39679 } }
I20260928 06:33:23.514693 18636 ts_tablet_manager.cc:1428] T e9596bb0ecd24bfcb9f91493eb23dfa1 P 8cf20e908f614fea9452dbe848d1dac3: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 06:33:23.514840 18641 tablet_replica.cc:439] T e9596bb0ecd24bfcb9f91493eb23dfa1 P 8cf20e908f614fea9452dbe848d1dac3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cf20e908f614fea9452dbe848d1dac3" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 35783 } } peers { permanent_uuid: "02a32df533b64624934cc9f57e59fc8b" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 33503 } } peers { permanent_uuid: "a5ca55a791204376a9ddd1bd72b4cff6" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39679 } } }
I20260928 06:33:23.514959 18641 tablet_replica.cc:442] T e9596bb0ecd24bfcb9f91493eb23dfa1 P 8cf20e908f614fea9452dbe848d1dac3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:33:23.762588 18637 raft_consensus.cc:482] T e9596bb0ecd24bfcb9f91493eb23dfa1 P a5ca55a791204376a9ddd1bd72b4cff6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:33:23.762699 18637 raft_consensus.cc:504] T e9596bb0ecd24bfcb9f91493eb23dfa1 P a5ca55a791204376a9ddd1bd72b4cff6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cf20e908f614fea9452dbe848d1dac3" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 35783 } } peers { permanent_uuid: "02a32df533b64624934cc9f57e59fc8b" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 33503 } } peers { permanent_uuid: "a5ca55a791204376a9ddd1bd72b4cff6" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39679 } }
I20260928 06:33:23.762871 18637 leader_election.cc:288] T e9596bb0ecd24bfcb9f91493eb23dfa1 P a5ca55a791204376a9ddd1bd72b4cff6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8cf20e908f614fea9452dbe848d1dac3 (127.16.157.65:35783), 02a32df533b64624934cc9f57e59fc8b (127.16.157.66:33503)
I20260928 06:33:23.763154 18285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9596bb0ecd24bfcb9f91493eb23dfa1" candidate_uuid: "a5ca55a791204376a9ddd1bd72b4cff6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cf20e908f614fea9452dbe848d1dac3" is_pre_election: true
I20260928 06:33:23.763199 18425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9596bb0ecd24bfcb9f91493eb23dfa1" candidate_uuid: "a5ca55a791204376a9ddd1bd72b4cff6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02a32df533b64624934cc9f57e59fc8b" is_pre_election: true
I20260928 06:33:23.763278 18285 raft_consensus.cc:2454] T e9596bb0ecd24bfcb9f91493eb23dfa1 P 8cf20e908f614fea9452dbe848d1dac3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5ca55a791204376a9ddd1bd72b4cff6 in term 0.
I20260928 06:33:23.763311 18425 raft_consensus.cc:2454] T e9596bb0ecd24bfcb9f91493eb23dfa1 P 02a32df533b64624934cc9f57e59fc8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5ca55a791204376a9ddd1bd72b4cff6 in term 0.
I20260928 06:33:23.763502 18498 leader_election.cc:302] T e9596bb0ecd24bfcb9f91493eb23dfa1 P a5ca55a791204376a9ddd1bd72b4cff6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02a32df533b64624934cc9f57e59fc8b, a5ca55a791204376a9ddd1bd72b4cff6; no voters:
I20260928 06:33:23.763602 18637 raft_consensus.cc:2781] T e9596bb0ecd24bfcb9f91493eb23dfa1 P a5ca55a791204376a9ddd1bd72b4cff6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:33:23.763643 18637 raft_consensus.cc:482] T e9596bb0ecd24bfcb9f91493eb23dfa1 P a5ca55a791204376a9ddd1bd72b4cff6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:33:23.763669 18637 raft_consensus.cc:3037] T e9596bb0ecd24bfcb9f91493eb23dfa1 P a5ca55a791204376a9ddd1bd72b4cff6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:23.764384 18637 raft_consensus.cc:504] T e9596bb0ecd24bfcb9f91493eb23dfa1 P a5ca55a791204376a9ddd1bd72b4cff6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cf20e908f614fea9452dbe848d1dac3" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 35783 } } peers { permanent_uuid: "02a32df533b64624934cc9f57e59fc8b" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 33503 } } peers { permanent_uuid: "a5ca55a791204376a9ddd1bd72b4cff6" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39679 } }
I20260928 06:33:23.764518 18637 leader_election.cc:288] T e9596bb0ecd24bfcb9f91493eb23dfa1 P a5ca55a791204376a9ddd1bd72b4cff6 [CANDIDATE]: Term 1 election: Requested vote from peers 8cf20e908f614fea9452dbe848d1dac3 (127.16.157.65:35783), 02a32df533b64624934cc9f57e59fc8b (127.16.157.66:33503)
I20260928 06:33:23.764734 18285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9596bb0ecd24bfcb9f91493eb23dfa1" candidate_uuid: "a5ca55a791204376a9ddd1bd72b4cff6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cf20e908f614fea9452dbe848d1dac3"
I20260928 06:33:23.764747 18425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9596bb0ecd24bfcb9f91493eb23dfa1" candidate_uuid: "a5ca55a791204376a9ddd1bd72b4cff6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02a32df533b64624934cc9f57e59fc8b"
I20260928 06:33:23.764832 18285 raft_consensus.cc:3037] T e9596bb0ecd24bfcb9f91493eb23dfa1 P 8cf20e908f614fea9452dbe848d1dac3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:23.764837 18425 raft_consensus.cc:3037] T e9596bb0ecd24bfcb9f91493eb23dfa1 P 02a32df533b64624934cc9f57e59fc8b [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:23.765443 18285 raft_consensus.cc:2454] T e9596bb0ecd24bfcb9f91493eb23dfa1 P 8cf20e908f614fea9452dbe848d1dac3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5ca55a791204376a9ddd1bd72b4cff6 in term 1.
I20260928 06:33:23.765632 18498 leader_election.cc:302] T e9596bb0ecd24bfcb9f91493eb23dfa1 P a5ca55a791204376a9ddd1bd72b4cff6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8cf20e908f614fea9452dbe848d1dac3, a5ca55a791204376a9ddd1bd72b4cff6; no voters:
I20260928 06:33:23.765683 18425 raft_consensus.cc:2454] T e9596bb0ecd24bfcb9f91493eb23dfa1 P 02a32df533b64624934cc9f57e59fc8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5ca55a791204376a9ddd1bd72b4cff6 in term 1.
I20260928 06:33:23.765748 18637 raft_consensus.cc:2781] T e9596bb0ecd24bfcb9f91493eb23dfa1 P a5ca55a791204376a9ddd1bd72b4cff6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:33:23.765795 18637 raft_consensus.cc:686] T e9596bb0ecd24bfcb9f91493eb23dfa1 P a5ca55a791204376a9ddd1bd72b4cff6 [term 1 LEADER]: Becoming Leader. State: Replica: a5ca55a791204376a9ddd1bd72b4cff6, State: Running, Role: LEADER
I20260928 06:33:23.765851 18637 consensus_queue.cc:237] T e9596bb0ecd24bfcb9f91493eb23dfa1 P a5ca55a791204376a9ddd1bd72b4cff6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cf20e908f614fea9452dbe848d1dac3" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 35783 } } peers { permanent_uuid: "02a32df533b64624934cc9f57e59fc8b" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 33503 } } peers { permanent_uuid: "a5ca55a791204376a9ddd1bd72b4cff6" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39679 } }
I20260928 06:33:23.766141 18664 tablet_replica.cc:439] T e9596bb0ecd24bfcb9f91493eb23dfa1 P a5ca55a791204376a9ddd1bd72b4cff6: TxnStatusTablet state changed. Reason: New leader a5ca55a791204376a9ddd1bd72b4cff6. Latest consensus state: current_term: 1 leader_uuid: "a5ca55a791204376a9ddd1bd72b4cff6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cf20e908f614fea9452dbe848d1dac3" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 35783 } } peers { permanent_uuid: "02a32df533b64624934cc9f57e59fc8b" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 33503 } } peers { permanent_uuid: "a5ca55a791204376a9ddd1bd72b4cff6" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39679 } } }
I20260928 06:33:23.766244 18664 tablet_replica.cc:442] T e9596bb0ecd24bfcb9f91493eb23dfa1 P a5ca55a791204376a9ddd1bd72b4cff6: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:33:23.766402 18676 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 06:33:23.766517 18676 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 06:33:23.766463 18128 catalog_manager.cc:5495] T e9596bb0ecd24bfcb9f91493eb23dfa1 P a5ca55a791204376a9ddd1bd72b4cff6 reported cstate change: term changed from 0 to 1, leader changed from <none> to a5ca55a791204376a9ddd1bd72b4cff6 (127.16.157.67). New cstate: current_term: 1 leader_uuid: "a5ca55a791204376a9ddd1bd72b4cff6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cf20e908f614fea9452dbe848d1dac3" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 35783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02a32df533b64624934cc9f57e59fc8b" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 33503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5ca55a791204376a9ddd1bd72b4cff6" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39679 } health_report { overall_health: HEALTHY } } }
I20260928 06:33:23.799918 18633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:23.797202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36374 (local address 127.16.157.67:39679)
0928 06:33:23.797277 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:23.797279 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:23.797290 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:33:23.797420 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:23.797421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:23.797443 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:23.797512 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:33:23.797699 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:23.798245 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:23.799055 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:23.799261 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:23.799350 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:23.799552 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:33:23.799665 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:33:23.799754 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:23.799811 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 06:33:23.799839 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 06:33:23.799960 18671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:23.797116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.67:39679 (local address 127.0.0.1:36374)
0928 06:33:23.797201 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:23.797232 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:23.797301 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:23.797528 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:23.797529 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:23.797530 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:33:23.797616 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:23.797618 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:23.798341 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:23.798342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:23.798957 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:23.798964 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:23.799348 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:23.799349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:23.799429 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:23.799450 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:33:23.799741 (+ 291us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:33:23.799751 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 06:33:23.799809 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 06:33:23.799885 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 06:33:23.801200 18425 raft_consensus.cc:1261] T e9596bb0ecd24bfcb9f91493eb23dfa1 P 02a32df533b64624934cc9f57e59fc8b [term 1 FOLLOWER]: Refusing update from remote peer a5ca55a791204376a9ddd1bd72b4cff6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:33:23.801559 18637 consensus_queue.cc:1035] T e9596bb0ecd24bfcb9f91493eb23dfa1 P a5ca55a791204376a9ddd1bd72b4cff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02a32df533b64624934cc9f57e59fc8b" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:33:23.802423 18644 tablet_replica.cc:439] T e9596bb0ecd24bfcb9f91493eb23dfa1 P 02a32df533b64624934cc9f57e59fc8b: TxnStatusTablet state changed. Reason: New leader a5ca55a791204376a9ddd1bd72b4cff6. Latest consensus state: current_term: 1 leader_uuid: "a5ca55a791204376a9ddd1bd72b4cff6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cf20e908f614fea9452dbe848d1dac3" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 35783 } } peers { permanent_uuid: "02a32df533b64624934cc9f57e59fc8b" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 33503 } } peers { permanent_uuid: "a5ca55a791204376a9ddd1bd72b4cff6" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39679 } } }
I20260928 06:33:23.802527 18644 tablet_replica.cc:442] T e9596bb0ecd24bfcb9f91493eb23dfa1 P 02a32df533b64624934cc9f57e59fc8b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:33:23.803011 18664 tablet_replica.cc:439] T e9596bb0ecd24bfcb9f91493eb23dfa1 P a5ca55a791204376a9ddd1bd72b4cff6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a5ca55a791204376a9ddd1bd72b4cff6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cf20e908f614fea9452dbe848d1dac3" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 35783 } } peers { permanent_uuid: "02a32df533b64624934cc9f57e59fc8b" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 33503 } } peers { permanent_uuid: "a5ca55a791204376a9ddd1bd72b4cff6" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39679 } } }
I20260928 06:33:23.803040 18285 raft_consensus.cc:1261] T e9596bb0ecd24bfcb9f91493eb23dfa1 P 8cf20e908f614fea9452dbe848d1dac3 [term 1 FOLLOWER]: Refusing update from remote peer a5ca55a791204376a9ddd1bd72b4cff6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:33:23.803104 18664 tablet_replica.cc:442] T e9596bb0ecd24bfcb9f91493eb23dfa1 P a5ca55a791204376a9ddd1bd72b4cff6: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:33:23.803177 18649 mvcc.cc:205] Tried to move back new op lower bound from 7334204226767257600 to 7334204226625277952. Current Snapshot: MvccSnapshot[applied={T|T < 7334204226767257600 or (T in {7334204226767257600})}]
I20260928 06:33:23.803458 18664 consensus_queue.cc:1035] T e9596bb0ecd24bfcb9f91493eb23dfa1 P a5ca55a791204376a9ddd1bd72b4cff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cf20e908f614fea9452dbe848d1dac3" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 35783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:33:23.803952 18647 mvcc.cc:205] Tried to move back new op lower bound from 7334204226767257600 to 7334204226625277952. Current Snapshot: MvccSnapshot[applied={T|T < 7334204226767257600}]
I20260928 06:33:23.804200 18644 tablet_replica.cc:439] T e9596bb0ecd24bfcb9f91493eb23dfa1 P 02a32df533b64624934cc9f57e59fc8b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a5ca55a791204376a9ddd1bd72b4cff6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cf20e908f614fea9452dbe848d1dac3" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 35783 } } peers { permanent_uuid: "02a32df533b64624934cc9f57e59fc8b" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 33503 } } peers { permanent_uuid: "a5ca55a791204376a9ddd1bd72b4cff6" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39679 } } }
I20260928 06:33:23.804279 18644 tablet_replica.cc:442] T e9596bb0ecd24bfcb9f91493eb23dfa1 P 02a32df533b64624934cc9f57e59fc8b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:33:23.804675 18641 tablet_replica.cc:439] T e9596bb0ecd24bfcb9f91493eb23dfa1 P 8cf20e908f614fea9452dbe848d1dac3: TxnStatusTablet state changed. Reason: New leader a5ca55a791204376a9ddd1bd72b4cff6. Latest consensus state: current_term: 1 leader_uuid: "a5ca55a791204376a9ddd1bd72b4cff6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cf20e908f614fea9452dbe848d1dac3" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 35783 } } peers { permanent_uuid: "02a32df533b64624934cc9f57e59fc8b" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 33503 } } peers { permanent_uuid: "a5ca55a791204376a9ddd1bd72b4cff6" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39679 } } }
I20260928 06:33:23.804781 18641 tablet_replica.cc:442] T e9596bb0ecd24bfcb9f91493eb23dfa1 P 8cf20e908f614fea9452dbe848d1dac3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:33:23.804828 18641 tablet_replica.cc:439] T e9596bb0ecd24bfcb9f91493eb23dfa1 P 8cf20e908f614fea9452dbe848d1dac3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a5ca55a791204376a9ddd1bd72b4cff6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cf20e908f614fea9452dbe848d1dac3" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 35783 } } peers { permanent_uuid: "02a32df533b64624934cc9f57e59fc8b" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 33503 } } peers { permanent_uuid: "a5ca55a791204376a9ddd1bd72b4cff6" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39679 } } }
I20260928 06:33:23.804878 18641 tablet_replica.cc:442] T e9596bb0ecd24bfcb9f91493eb23dfa1 P 8cf20e908f614fea9452dbe848d1dac3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:33:23.807480 18633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:23.804709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36382 (local address 127.16.157.67:39679)
0928 06:33:23.805119 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:23.805121 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:23.805130 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:33:23.805150 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:23.805151 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:23.805162 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:23.805218 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:33:23.805384 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:23.805983 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:23.806736 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:23.806949 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:23.807032 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:23.807185 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:33:23.807252 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:33:23.807314 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:23.807376 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 06:33:23.807399 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388}
I20260928 06:33:23.813200 18633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:23.810316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:45507 (local address 127.16.157.67:39679)
0928 06:33:23.810376 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:23.810378 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:23.810529 (+ 151us) server_negotiation.cc:407] Connection header received
0928 06:33:23.810585 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:23.810586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:23.810594 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:23.810635 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:33:23.810786 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:23.811319 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:23.812486 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:23.812660 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:23.812734 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:23.812936 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:33:23.812961 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:33:23.812997 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:23.813081 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 06:33:23.813116 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":14,"server-negotiator.queue_time_us":37}
I20260928 06:33:23.813288 18684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:23.810235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.67:39679 (local address 127.16.157.67:45507)
0928 06:33:23.810445 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:23.810456 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:23.810528 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:23.810649 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:23.810650 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:23.810652 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:33:23.810729 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:23.810730 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:23.811400 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:23.811401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:23.812407 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:23.812410 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:23.812768 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:23.812770 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:23.812836 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:23.812852 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:33:23.813025 (+ 173us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:33:23.813026 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 06:33:23.813079 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 06:33:23.813135 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":2,"thread_start_us":107,"threads_started":1}
I20260928 06:33:23.814016 18664 tablet_replica.cc:439] T e9596bb0ecd24bfcb9f91493eb23dfa1 P a5ca55a791204376a9ddd1bd72b4cff6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a5ca55a791204376a9ddd1bd72b4cff6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cf20e908f614fea9452dbe848d1dac3" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 35783 } } peers { permanent_uuid: "02a32df533b64624934cc9f57e59fc8b" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 33503 } } peers { permanent_uuid: "a5ca55a791204376a9ddd1bd72b4cff6" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39679 } } }
I20260928 06:33:23.814117 18664 tablet_replica.cc:442] T e9596bb0ecd24bfcb9f91493eb23dfa1 P a5ca55a791204376a9ddd1bd72b4cff6: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:33:23.815801 18425 raft_consensus.cc:1261] T 7df6f425df9a44c4bcf27c37e13d708b P 02a32df533b64624934cc9f57e59fc8b [term 1 FOLLOWER]: Refusing update from remote peer a5ca55a791204376a9ddd1bd72b4cff6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:33:23.815866 18285 raft_consensus.cc:1261] T 7df6f425df9a44c4bcf27c37e13d708b P 8cf20e908f614fea9452dbe848d1dac3 [term 1 FOLLOWER]: Refusing update from remote peer a5ca55a791204376a9ddd1bd72b4cff6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:33:23.816160 18685 consensus_queue.cc:1035] T 7df6f425df9a44c4bcf27c37e13d708b P a5ca55a791204376a9ddd1bd72b4cff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cf20e908f614fea9452dbe848d1dac3" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 35783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:33:23.816267 18664 consensus_queue.cc:1035] T 7df6f425df9a44c4bcf27c37e13d708b P a5ca55a791204376a9ddd1bd72b4cff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02a32df533b64624934cc9f57e59fc8b" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:33:23.831478 18127 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:33:23.831609 18127 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:33:23.832296 18127 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ee1eda9c7d684f88a77cb007b5502e30: Sending DeleteTablet for 3 replicas of tablet 7df6f425df9a44c4bcf27c37e13d708b
I20260928 06:33:23.832581 18544 tablet_service.cc:1514] Processing DeleteTablet for tablet 7df6f425df9a44c4bcf27c37e13d708b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:33:23 UTC) from {username='kudu', principal='kudu/127.16.157.126@KRBTEST.COM'} at 127.0.0.1:36364
I20260928 06:33:23.832623 18405 tablet_service.cc:1514] Processing DeleteTablet for tablet 7df6f425df9a44c4bcf27c37e13d708b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:33:23 UTC) from {username='kudu', principal='kudu/127.16.157.126@KRBTEST.COM'} at 127.0.0.1:54412
I20260928 06:33:23.832693 18657 tablet_replica.cc:330] T 7df6f425df9a44c4bcf27c37e13d708b P a5ca55a791204376a9ddd1bd72b4cff6: stopping tablet replica
I20260928 06:33:23.832751 18657 raft_consensus.cc:2229] T 7df6f425df9a44c4bcf27c37e13d708b P a5ca55a791204376a9ddd1bd72b4cff6 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:33:23.832779 18659 tablet_replica.cc:330] T 7df6f425df9a44c4bcf27c37e13d708b P 02a32df533b64624934cc9f57e59fc8b: stopping tablet replica
I20260928 06:33:23.832849 18657 raft_consensus.cc:2258] T 7df6f425df9a44c4bcf27c37e13d708b P a5ca55a791204376a9ddd1bd72b4cff6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:33:23.832846 18659 raft_consensus.cc:2229] T 7df6f425df9a44c4bcf27c37e13d708b P 02a32df533b64624934cc9f57e59fc8b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:33:23.832888 18659 raft_consensus.cc:2258] T 7df6f425df9a44c4bcf27c37e13d708b P 02a32df533b64624934cc9f57e59fc8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:33:23.832984 18265 tablet_service.cc:1514] Processing DeleteTablet for tablet 7df6f425df9a44c4bcf27c37e13d708b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:33:23 UTC) from {username='kudu', principal='kudu/127.16.157.126@KRBTEST.COM'} at 127.0.0.1:41740
I20260928 06:33:23.833081 18658 tablet_replica.cc:330] T 7df6f425df9a44c4bcf27c37e13d708b P 8cf20e908f614fea9452dbe848d1dac3: stopping tablet replica
I20260928 06:33:23.833146 18658 raft_consensus.cc:2229] T 7df6f425df9a44c4bcf27c37e13d708b P 8cf20e908f614fea9452dbe848d1dac3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:33:23.833196 18658 raft_consensus.cc:2258] T 7df6f425df9a44c4bcf27c37e13d708b P 8cf20e908f614fea9452dbe848d1dac3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:33:23.833190 18657 ts_tablet_manager.cc:1905] T 7df6f425df9a44c4bcf27c37e13d708b P a5ca55a791204376a9ddd1bd72b4cff6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:33:23.833304 18659 ts_tablet_manager.cc:1905] T 7df6f425df9a44c4bcf27c37e13d708b P 02a32df533b64624934cc9f57e59fc8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:33:23.837903 18658 ts_tablet_manager.cc:1905] T 7df6f425df9a44c4bcf27c37e13d708b P 8cf20e908f614fea9452dbe848d1dac3: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 06:33:23 dist-test-slave-rrj5 krb5kdc[18083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577203, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.157.65@KRBTEST.COM
I20260928 06:33:23.843649 18631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:23.833462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41746 (local address 127.16.157.65:35783)
0928 06:33:23.833506 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:23.833508 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:23.834498 (+ 990us) server_negotiation.cc:407] Connection header received
0928 06:33:23.834545 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:23.834545 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:23.834559 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:23.834611 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:23.835291 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:23.835805 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:23.836756 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:23.836937 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:23.837014 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:23.840355 (+ 3341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:23.840367 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:23.840368 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:23.840376 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:23.842189 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:23.842690 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:23.842693 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:23.842694 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:23.842785 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:23.843024 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:23.843025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:23.843025 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:23.843164 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:23.843275 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:23.843421 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 06:33:23.843530 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 06:33:23.844000 18319 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41746
I20260928 06:33:23.851703 18178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:23.845116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33266 (local address 127.16.157.126:40707)
0928 06:33:23.845165 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:23.845167 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:23.845226 (+ 59us) server_negotiation.cc:407] Connection header received
0928 06:33:23.845297 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:23.845298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:23.845308 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:23.845358 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:23.845977 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:23.846549 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:23.847331 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:23.847596 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:23.847672 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:23.848933 (+ 1261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:23.848934 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:23.848935 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:23.848943 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:23.850532 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:23.850923 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:23.850924 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:23.850925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:23.850986 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:23.851171 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:23.851172 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:23.851172 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:23.851301 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:23.851412 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:23.851511 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 06:33:23.851593 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 06:33:23.852178 18127 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33266
I20260928 06:33:23.852846 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 18200
I20260928 06:33:23.858098 18657 ts_tablet_manager.cc:1918] T 7df6f425df9a44c4bcf27c37e13d708b P a5ca55a791204376a9ddd1bd72b4cff6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:33:23.858098 18659 ts_tablet_manager.cc:1918] T 7df6f425df9a44c4bcf27c37e13d708b P 02a32df533b64624934cc9f57e59fc8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:33:23.858175 18659 log.cc:1196] T 7df6f425df9a44c4bcf27c37e13d708b P 02a32df533b64624934cc9f57e59fc8b: Deleting WAL directory at /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/ts-1/wal/wals/7df6f425df9a44c4bcf27c37e13d708b
I20260928 06:33:23.858175 18657 log.cc:1196] T 7df6f425df9a44c4bcf27c37e13d708b P a5ca55a791204376a9ddd1bd72b4cff6: Deleting WAL directory at /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577196946045-17013-0/minicluster-data/ts-2/wal/wals/7df6f425df9a44c4bcf27c37e13d708b
I20260928 06:33:23.858418 18659 ts_tablet_manager.cc:1939] T 7df6f425df9a44c4bcf27c37e13d708b P 02a32df533b64624934cc9f57e59fc8b: Deleting consensus metadata
I20260928 06:33:23.858418 18657 ts_tablet_manager.cc:1939] T 7df6f425df9a44c4bcf27c37e13d708b P a5ca55a791204376a9ddd1bd72b4cff6: Deleting consensus metadata
W20260928 06:33:23.861845 18114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.157.65:35783 (error 108)
I20260928 06:33:23.862106 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 18340
I20260928 06:33:23.863884 18694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:23.862961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.65:35783 (local address 127.0.0.1:41756)
0928 06:33:23.863762 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:23.863819 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.65:35783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":779,"thread_start_us":60,"threads_started":1}
W20260928 06:33:23.864059 18114 catalog_manager.cc:4584] TS 8cf20e908f614fea9452dbe848d1dac3 (127.16.157.65:35783): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7df6f425df9a44c4bcf27c37e13d708b: Network error: Client connection negotiation failed: client connection to 127.16.157.65:35783: connect: Connection refused (error 111)
I20260928 06:33:23.868880 18113 catalog_manager.cc:4841] TS a5ca55a791204376a9ddd1bd72b4cff6 (127.16.157.67:39679): tablet 7df6f425df9a44c4bcf27c37e13d708b (table test-table [id=ce1761ade9004ad9ba8a496b998d1ed2]) successfully deleted
I20260928 06:33:23.873327 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 18480
I20260928 06:33:23.873337 18694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:23.873185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.66:33503 (local address 127.0.0.1:54430)
0928 06:33:23.873255 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:23.873291 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.66:33503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 06:33:23.878820 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 18096
2026-09-28T06:33:23Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2782 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:33:23 dist-test-slave-rrj5 krb5kdc[18697](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:33:23 dist-test-slave-rrj5 krb5kdc[18697](info): set up 2 sockets
Sep 28 06:33:23 dist-test-slave-rrj5 krb5kdc[18697](info): commencing operation
krb5kdc: starting...
W20260928 06:33:24.974078 17013 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Sep 28 06:33:25 dist-test-slave-rrj5 krb5kdc[18697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:33:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:33:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.157.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab.
Entry for principal kudu/127.16.157.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab.
I20260928 06:33:25.100175 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:40157
--webserver_interface=127.16.157.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=kudu/127.16.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:34903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.157.126:40157
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:25.182801 18710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:25.183049 18710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:25.183087 18710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:25.184255 18710 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:33:25.184322 18710 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 06:33:25.184348 18710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:25.184381 18710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:25.184415 18710 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:33:25.184441 18710 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:33:25.186194 18710 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:34903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.157.126:40157
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=kudu/127.16.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:40157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:25.186491 18710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:25.186725 18710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:25.191627 18715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:25.191726 18716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:25.191623 18718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:25.192044 18710 server_base.cc:1034] running on GCE node
I20260928 06:33:25.192328 18710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:25.192641 18710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:25.193799 18710 hybrid_clock.cc:648] HybridClock initialized: now 1790577205193786 us; error 28 us; skew 500 ppm
Sep 28 06:33:25 dist-test-slave-rrj5 krb5kdc[18697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577205, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:25.196489 18710 init.cc:377] Logged in from keytab as kudu/127.16.157.126@KRBTEST.COM (short username kudu)
I20260928 06:33:25.196975 18710 webserver.cc:460] Webserver started at http://127.16.157.126:33307/ using document root <none> and password file <none>
I20260928 06:33:25.197208 18710 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:25.197285 18710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:25.197443 18710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:25.198589 18710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/master-0/data/instance:
uuid: "d9646455aab14be695ee9033c14d8ac0"
format_stamp: "Formatted at 2026-09-28 06:33:25 on dist-test-slave-rrj5"
server_key: "a333c1b77051fa174f1c1bbb57c59090"
server_key_iv: "d6f8a0ef8db0a6c5149307f2f0610005"
server_key_version: "encryptionkey@0"
I20260928 06:33:25.198974 18710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/master-0/wal/instance:
uuid: "d9646455aab14be695ee9033c14d8ac0"
format_stamp: "Formatted at 2026-09-28 06:33:25 on dist-test-slave-rrj5"
server_key: "a333c1b77051fa174f1c1bbb57c59090"
server_key_iv: "d6f8a0ef8db0a6c5149307f2f0610005"
server_key_version: "encryptionkey@0"
I20260928 06:33:25.200510 18710 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:33:25.201350 18725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:25.201526 18710 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:25.201663 18710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/master-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/master-0/wal
uuid: "d9646455aab14be695ee9033c14d8ac0"
format_stamp: "Formatted at 2026-09-28 06:33:25 on dist-test-slave-rrj5"
server_key: "a333c1b77051fa174f1c1bbb57c59090"
server_key_iv: "d6f8a0ef8db0a6c5149307f2f0610005"
server_key_version: "encryptionkey@0"
I20260928 06:33:25.201740 18710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:25.210963 18710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:25.213074 18710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:25.213255 18710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:25.218897 18710 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.126:40157
I20260928 06:33:25.218986 18787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.126:40157 every 8 connection(s)
I20260928 06:33:25.219357 18710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
I20260928 06:33:25.220064 18788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:25.222465 18788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9646455aab14be695ee9033c14d8ac0: Bootstrap starting.
I20260928 06:33:25.223080 18788 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9646455aab14be695ee9033c14d8ac0: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:25.223378 18788 log.cc:824] T 00000000000000000000000000000000 P d9646455aab14be695ee9033c14d8ac0: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:25.224131 18788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9646455aab14be695ee9033c14d8ac0: No bootstrap required, opened a new log
I20260928 06:33:25.225484 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 18710
I20260928 06:33:25.225569 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/master-0/wal/instance
I20260928 06:33:25.225803 17013 external_mini_cluster.cc:1428] Setting key 8919eb9d5a7bd03d653631917defbaba
I20260928 06:33:25.226709 18788 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9646455aab14be695ee9033c14d8ac0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9646455aab14be695ee9033c14d8ac0" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40157 } }
I20260928 06:33:25.226953 18788 raft_consensus.cc:374] T 00000000000000000000000000000000 P d9646455aab14be695ee9033c14d8ac0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:25.227005 18788 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9646455aab14be695ee9033c14d8ac0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9646455aab14be695ee9033c14d8ac0, State: Initialized, Role: FOLLOWER
I20260928 06:33:25.227151 18788 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9646455aab14be695ee9033c14d8ac0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9646455aab14be695ee9033c14d8ac0" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40157 } }
I20260928 06:33:25.227248 18788 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9646455aab14be695ee9033c14d8ac0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:33:25.227305 18788 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9646455aab14be695ee9033c14d8ac0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:33:25.227352 18788 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9646455aab14be695ee9033c14d8ac0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:25.227918 18788 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9646455aab14be695ee9033c14d8ac0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9646455aab14be695ee9033c14d8ac0" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40157 } }
I20260928 06:33:25.228041 18788 leader_election.cc:302] T 00000000000000000000000000000000 P d9646455aab14be695ee9033c14d8ac0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9646455aab14be695ee9033c14d8ac0; no voters:
I20260928 06:33:25.228245 18788 leader_election.cc:288] T 00000000000000000000000000000000 P d9646455aab14be695ee9033c14d8ac0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:33:25.228279 18793 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9646455aab14be695ee9033c14d8ac0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:33:25.228523 18793 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9646455aab14be695ee9033c14d8ac0 [term 1 LEADER]: Becoming Leader. State: Replica: d9646455aab14be695ee9033c14d8ac0, State: Running, Role: LEADER
I20260928 06:33:25.228592 18788 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9646455aab14be695ee9033c14d8ac0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:33:25.228698 18793 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9646455aab14be695ee9033c14d8ac0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9646455aab14be695ee9033c14d8ac0" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40157 } }
I20260928 06:33:25.229113 18793 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9646455aab14be695ee9033c14d8ac0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d9646455aab14be695ee9033c14d8ac0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9646455aab14be695ee9033c14d8ac0" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40157 } } }
I20260928 06:33:25.229291 18793 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9646455aab14be695ee9033c14d8ac0 [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:25.229482 18793 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9646455aab14be695ee9033c14d8ac0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9646455aab14be695ee9033c14d8ac0. Latest consensus state: current_term: 1 leader_uuid: "d9646455aab14be695ee9033c14d8ac0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9646455aab14be695ee9033c14d8ac0" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40157 } } }
I20260928 06:33:25.229650 18793 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9646455aab14be695ee9033c14d8ac0 [sys.catalog]: This master's current role is: LEADER
W20260928 06:33:25.230548 18808 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d9646455aab14be695ee9033c14d8ac0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:33:25.230640 18808 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:33:25.230942 18802 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:33:25.231109 18802 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 06:33:25 dist-test-slave-rrj5 krb5kdc[18697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:33:25.238251 18802 catalog_manager.cc:1347] Generated new cluster ID: d050ce8a762c47c3803ce87629f69977
I20260928 06:33:25.238320 18802 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:33:25.238886 18792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:25.226462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49194 (local address 127.16.157.126:40157)
0928 06:33:25.226731 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:25.226742 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:25.226779 (+ 37us) server_negotiation.cc:407] Connection header received
0928 06:33:25.227009 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:25.227019 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:25.227089 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:25.227241 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:25.228109 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:25.229408 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:25.230563 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:25.231192 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:25.231302 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:25.234292 (+ 2990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:25.234297 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:25.234297 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:25.234310 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:25.236461 (+ 2151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:25.236902 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:25.236906 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:25.236906 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:25.237069 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:25.237352 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:25.237353 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:25.237354 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:25.237725 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:25.237912 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:25.238498 (+ 586us) server_negotiation.cc:299] Negotiation successful
0928 06:33:25.238649 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":105,"threads_started":1}
I20260928 06:33:25.253329 18802 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:33:25.254070 18802 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:33:25.256536 18802 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d9646455aab14be695ee9033c14d8ac0: Generated new TSK 0
I20260928 06:33:25.256726 18802 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.157.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab.
Entry for principal kudu/127.16.157.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab.
I20260928 06:33:25.288543 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.65:0
--local_ip_for_outbound_sockets=127.16.157.65
--webserver_interface=127.16.157.65
--webserver_port=0
--tserver_master_addrs=127.16.157.126:40157
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab
--principal=kudu/127.16.157.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:34903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:25.408694 18814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:25.408988 18814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:25.409025 18814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:25.410859 18814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:25.410921 18814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:25.410952 18814 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 06:33:25.410970 18814 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:33:25.410996 18814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.65
I20260928 06:33:25.413218 18814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:34903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab
--principal=kudu/127.16.157.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.157.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.157.126:40157
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.65
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:25.413471 18814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:25.413761 18814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:25.420023 18822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:25.420167 18819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:25.420032 18820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:25.420389 18814 server_base.cc:1034] running on GCE node
I20260928 06:33:25.420603 18814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:25.420866 18814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:25.422029 18814 hybrid_clock.cc:648] HybridClock initialized: now 1790577205422008 us; error 36 us; skew 500 ppm
Sep 28 06:33:25 dist-test-slave-rrj5 krb5kdc[18697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577205, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:25.424734 18814 init.cc:377] Logged in from keytab as kudu/127.16.157.65@KRBTEST.COM (short username kudu)
I20260928 06:33:25.425215 18814 webserver.cc:460] Webserver started at http://127.16.157.65:35179/ using document root <none> and password file <none>
I20260928 06:33:25.425472 18814 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:25.425554 18814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:25.425758 18814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:25.426867 18814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/ts-0/data/instance:
uuid: "25aa964a114141aeb4b2374eaf7cb85f"
format_stamp: "Formatted at 2026-09-28 06:33:25 on dist-test-slave-rrj5"
server_key: "6a255cd1ba376bb8702b52e850dac832"
server_key_iv: "c1059b42a2eca9ce503bcb2f90b156e8"
server_key_version: "encryptionkey@0"
I20260928 06:33:25.427289 18814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/ts-0/wal/instance:
uuid: "25aa964a114141aeb4b2374eaf7cb85f"
format_stamp: "Formatted at 2026-09-28 06:33:25 on dist-test-slave-rrj5"
server_key: "6a255cd1ba376bb8702b52e850dac832"
server_key_iv: "c1059b42a2eca9ce503bcb2f90b156e8"
server_key_version: "encryptionkey@0"
I20260928 06:33:25.428766 18814 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:33:25.429597 18829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:25.429756 18814 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:25.429862 18814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/ts-0/wal
uuid: "25aa964a114141aeb4b2374eaf7cb85f"
format_stamp: "Formatted at 2026-09-28 06:33:25 on dist-test-slave-rrj5"
server_key: "6a255cd1ba376bb8702b52e850dac832"
server_key_iv: "c1059b42a2eca9ce503bcb2f90b156e8"
server_key_version: "encryptionkey@0"
I20260928 06:33:25.429946 18814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:25.440933 18814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:25.443148 18814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:25.443325 18814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:25.443910 18814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:33:25.443971 18814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:25.444011 18814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:33:25.444032 18814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 06:33:25 dist-test-slave-rrj5 krb5kdc[18697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577205, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.65@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:33:25.451829 18814 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.65:46219
I20260928 06:33:25.452095 18944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.65:46219 every 8 connection(s)
I20260928 06:33:25.452255 18814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
I20260928 06:33:25.455206 18792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:25.444612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:48485 (local address 127.16.157.126:40157)
0928 06:33:25.444691 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:25.444694 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:25.444707 (+ 13us) server_negotiation.cc:407] Connection header received
0928 06:33:25.445642 (+ 935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:25.445643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:25.445661 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:25.445716 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:25.446595 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:25.447449 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:25.449016 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:25.449379 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:25.449457 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:25.452129 (+ 2672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:25.452131 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:25.452131 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:25.452139 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:25.453760 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:25.454263 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:25.454264 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:25.454264 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:25.454320 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:25.454597 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:25.454598 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:25.454598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:25.454703 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:25.454754 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:25.455077 (+ 323us) server_negotiation.cc:299] Negotiation successful
0928 06:33:25.455143 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 06:33:25.455544 18838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:25.444311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40157 (local address 127.16.157.65:48485)
0928 06:33:25.444506 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:25.444561 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:25.445478 (+ 917us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:25.445852 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:25.445857 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:25.446209 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:25.446491 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:25.446500 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:25.447561 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:25.447562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:25.448912 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:25.448915 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:25.449470 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:25.449472 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:25.449659 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:25.450263 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:25.450273 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:25.452009 (+ 1736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:25.453834 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:25.453841 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:25.453843 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:25.454179 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:25.454384 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:25.454385 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:25.454385 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:25.454530 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:25.454769 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:25.454775 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:25.454941 (+ 166us) client_negotiation.cc:769] Sending connection context
0928 06:33:25.455057 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 06:33:25.455294 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":89,"threads_started":1}
I20260928 06:33:25.455729 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 18814
I20260928 06:33:25.455832 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/ts-0/wal/instance
I20260928 06:33:25.456037 17013 external_mini_cluster.cc:1428] Setting key 400f76fb901d41925a0178c27af0e218
I20260928 06:33:25.456254 18945 heartbeater.cc:347] Connected to a master server at 127.16.157.126:40157
I20260928 06:33:25.456346 18945 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:25.456532 18945 heartbeater.cc:511] Master 127.16.157.126:40157 requested a full tablet report, sending...
I20260928 06:33:25.457077 18742 ts_manager.cc:194] Registered new tserver with Master: 25aa964a114141aeb4b2374eaf7cb85f (127.16.157.65:46219)
I20260928 06:33:25.457965 18742 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.157.65@KRBTEST.COM'} at 127.16.157.65:48485
I20260928 06:33:25.463909 18792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:25.457209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:38757 (local address 127.16.157.126:40157)
0928 06:33:25.457247 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:25.457249 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:25.457481 (+ 232us) server_negotiation.cc:407] Connection header received
0928 06:33:25.457553 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:25.457554 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:25.457563 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:25.457631 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:25.458022 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:25.458761 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:25.460033 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:25.460236 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:25.460348 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:25.461312 (+ 964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:25.461314 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:25.461314 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:25.461321 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:25.462744 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:25.463115 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:25.463116 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:25.463117 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:25.463176 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:25.463388 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:25.463389 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:25.463389 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:25.463498 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:25.463569 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:25.463733 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 06:33:25.463815 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:33:25.463915 18951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:25.457278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40157 (local address 127.16.157.65:38757)
0928 06:33:25.457401 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:25.457411 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:25.457494 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:25.457685 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:25.457686 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:25.457875 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:25.457939 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:25.457940 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:25.459043 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:25.459044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:25.459941 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:25.459944 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:25.460345 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:25.460346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:25.460436 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:25.460898 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:25.460903 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:25.461196 (+ 293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:25.462849 (+ 1653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:25.462850 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:25.462850 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:25.463027 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:25.463249 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:25.463250 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:25.463251 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:25.463317 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:25.463573 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:25.463574 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:25.463618 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 06:33:25.463706 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 06:33:25.463817 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":76,"threads_started":1}
WARNING: no policy specified for kudu/127.16.157.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab.
Entry for principal kudu/127.16.157.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab.
I20260928 06:33:25.486562 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.66:0
--local_ip_for_outbound_sockets=127.16.157.66
--webserver_interface=127.16.157.66
--webserver_port=0
--tserver_master_addrs=127.16.157.126:40157
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab
--principal=kudu/127.16.157.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:34903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:25.596532 18954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:25.596743 18954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:25.596769 18954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:25.598027 18954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:25.598070 18954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:25.598091 18954 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 06:33:25.598104 18954 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:33:25.598160 18954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.66
I20260928 06:33:25.599633 18954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:34903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab
--principal=kudu/127.16.157.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.157.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.157.126:40157
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.66
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:25.599891 18954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:25.600170 18954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:25.605360 18962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:25.605350 18959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:25.605350 18960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:25.606084 18954 server_base.cc:1034] running on GCE node
I20260928 06:33:25.606292 18954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:25.606536 18954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:25.607702 18954 hybrid_clock.cc:648] HybridClock initialized: now 1790577205607688 us; error 31 us; skew 500 ppm
Sep 28 06:33:25 dist-test-slave-rrj5 krb5kdc[18697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577205, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:25.610217 18954 init.cc:377] Logged in from keytab as kudu/127.16.157.66@KRBTEST.COM (short username kudu)
I20260928 06:33:25.610707 18954 webserver.cc:460] Webserver started at http://127.16.157.66:41909/ using document root <none> and password file <none>
I20260928 06:33:25.610977 18954 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:25.611045 18954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:25.611193 18954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:25.612287 18954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/ts-1/data/instance:
uuid: "bffb4fdc681a4dd39a816b4a9706a7af"
format_stamp: "Formatted at 2026-09-28 06:33:25 on dist-test-slave-rrj5"
server_key: "4506fa2285ee0840090419d43c9965c3"
server_key_iv: "c4457c37e43a39892129b44fae925293"
server_key_version: "encryptionkey@0"
I20260928 06:33:25.612634 18954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/ts-1/wal/instance:
uuid: "bffb4fdc681a4dd39a816b4a9706a7af"
format_stamp: "Formatted at 2026-09-28 06:33:25 on dist-test-slave-rrj5"
server_key: "4506fa2285ee0840090419d43c9965c3"
server_key_iv: "c4457c37e43a39892129b44fae925293"
server_key_version: "encryptionkey@0"
I20260928 06:33:25.614053 18954 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 06:33:25.614849 18969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:25.615013 18954 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:25.615123 18954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/ts-1/wal
uuid: "bffb4fdc681a4dd39a816b4a9706a7af"
format_stamp: "Formatted at 2026-09-28 06:33:25 on dist-test-slave-rrj5"
server_key: "4506fa2285ee0840090419d43c9965c3"
server_key_iv: "c4457c37e43a39892129b44fae925293"
server_key_version: "encryptionkey@0"
I20260928 06:33:25.615214 18954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:25.632977 18954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:25.635258 18954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:25.635406 18954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:25.636032 18954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:33:25.636082 18954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:25.636116 18954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:33:25.636140 18954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 06:33:25 dist-test-slave-rrj5 krb5kdc[18697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577205, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.66@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:33:25.643700 18954 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.66:41079
I20260928 06:33:25.643746 19084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.66:41079 every 8 connection(s)
I20260928 06:33:25.644129 18954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
I20260928 06:33:25.646952 18792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:25.636728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.66:44711 (local address 127.16.157.126:40157)
0928 06:33:25.636780 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:25.636782 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:25.636795 (+ 13us) server_negotiation.cc:407] Connection header received
0928 06:33:25.637356 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:25.637357 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:25.637372 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:25.637445 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:25.638308 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:25.639182 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:25.640922 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:25.641266 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:25.641344 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:25.644126 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:25.644128 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:25.644128 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:25.644137 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:25.645672 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:25.646060 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:25.646061 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:25.646061 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:25.646125 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:25.646340 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:25.646341 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:25.646341 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:25.646476 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:25.646551 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:25.646779 (+ 228us) server_negotiation.cc:299] Negotiation successful
0928 06:33:25.646864 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 06:33:25.647060 18978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:25.636389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40157 (local address 127.16.157.66:44711)
0928 06:33:25.636565 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:25.636620 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:25.637220 (+ 600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:25.637525 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:25.637529 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:25.637898 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:25.638162 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:25.638168 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:25.639305 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:25.639306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:25.640808 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:25.640811 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:25.641356 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:25.641357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:25.641511 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:25.642141 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:25.642152 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:25.644019 (+ 1867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:25.645745 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:25.645751 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:25.645753 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:25.645971 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:25.646186 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:25.646187 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:25.646187 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:25.646281 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:25.646536 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:25.646539 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:25.646645 (+ 106us) client_negotiation.cc:769] Sending connection context
0928 06:33:25.646737 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 06:33:25.646861 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":87,"threads_started":1}
I20260928 06:33:25.647512 19085 heartbeater.cc:347] Connected to a master server at 127.16.157.126:40157
I20260928 06:33:25.647593 19085 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:25.647764 19085 heartbeater.cc:511] Master 127.16.157.126:40157 requested a full tablet report, sending...
I20260928 06:33:25.648175 18741 ts_manager.cc:194] Registered new tserver with Master: bffb4fdc681a4dd39a816b4a9706a7af (127.16.157.66:41079)
I20260928 06:33:25.648655 18741 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.157.66@KRBTEST.COM'} at 127.16.157.66:44711
I20260928 06:33:25.652756 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 18954
I20260928 06:33:25.652864 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/ts-1/wal/instance
I20260928 06:33:25.653051 17013 external_mini_cluster.cc:1428] Setting key 6f2cd008afc4226a232e33fe16b34fe9
I20260928 06:33:25.655107 19091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:25.648984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40157 (local address 127.16.157.66:42337)
0928 06:33:25.649115 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:25.649125 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:25.649191 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:25.649349 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:25.649350 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:25.649563 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:25.649649 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:25.649651 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:25.650311 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:25.650312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:25.650851 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:25.650853 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:25.651224 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:25.651225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:25.651302 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:25.651766 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:25.651772 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:25.652079 (+ 307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:25.653937 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:25.653938 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:25.653939 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:25.654142 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:25.654357 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:25.654357 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:25.654358 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:25.654423 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:25.654769 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:25.654770 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:25.654835 (+ 65us) client_negotiation.cc:769] Sending connection context
0928 06:33:25.654906 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 06:33:25.655014 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":72,"threads_started":1}
I20260928 06:33:25.655167 18792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:25.649036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.66:42337 (local address 127.16.157.126:40157)
0928 06:33:25.649075 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:25.649077 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:25.649192 (+ 115us) server_negotiation.cc:407] Connection header received
0928 06:33:25.649233 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:25.649234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:25.649244 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:25.649303 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:25.649727 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:25.650215 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:25.650933 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:25.651139 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:25.651223 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:25.652189 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:25.652190 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:25.652191 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:25.652197 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:25.653846 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:25.654223 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:25.654224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:25.654224 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:25.654285 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:25.654488 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:25.654489 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:25.654489 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:25.654696 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:25.654765 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:25.654946 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 06:33:25.655033 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
WARNING: no policy specified for kudu/127.16.157.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab.
Entry for principal kudu/127.16.157.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab.
I20260928 06:33:25.678586 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.67:0
--local_ip_for_outbound_sockets=127.16.157.67
--webserver_interface=127.16.157.67
--webserver_port=0
--tserver_master_addrs=127.16.157.126:40157
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab
--principal=kudu/127.16.157.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:34903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:25.762180 19094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:25.762384 19094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:25.762409 19094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:25.763589 19094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:25.763628 19094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:25.763649 19094 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 06:33:25.763664 19094 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:33:25.763703 19094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.67
I20260928 06:33:25.765173 19094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:34903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab
--principal=kudu/127.16.157.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.157.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.157.126:40157
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.67
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:25.765414 19094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:25.765682 19094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:25.770587 19100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:25.770709 19102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:25.770745 19094 server_base.cc:1034] running on GCE node
W20260928 06:33:25.770606 19099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:25.771061 19094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:25.771332 19094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:25.772485 19094 hybrid_clock.cc:648] HybridClock initialized: now 1790577205772470 us; error 30 us; skew 500 ppm
Sep 28 06:33:25 dist-test-slave-rrj5 krb5kdc[18697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577205, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:25.775019 19094 init.cc:377] Logged in from keytab as kudu/127.16.157.67@KRBTEST.COM (short username kudu)
I20260928 06:33:25.775547 19094 webserver.cc:460] Webserver started at http://127.16.157.67:44875/ using document root <none> and password file <none>
I20260928 06:33:25.775781 19094 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:25.775825 19094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:25.775964 19094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:25.777002 19094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/ts-2/data/instance:
uuid: "5cbf45ff2943424594f6a7568d3a93de"
format_stamp: "Formatted at 2026-09-28 06:33:25 on dist-test-slave-rrj5"
server_key: "1e8ae9c7e770cfb5a0b4e6c7c3d46ebd"
server_key_iv: "e9dfa99bb1dcf7c70fd698b27883b499"
server_key_version: "encryptionkey@0"
I20260928 06:33:25.777395 19094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/ts-2/wal/instance:
uuid: "5cbf45ff2943424594f6a7568d3a93de"
format_stamp: "Formatted at 2026-09-28 06:33:25 on dist-test-slave-rrj5"
server_key: "1e8ae9c7e770cfb5a0b4e6c7c3d46ebd"
server_key_iv: "e9dfa99bb1dcf7c70fd698b27883b499"
server_key_version: "encryptionkey@0"
I20260928 06:33:25.778772 19094 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:33:25.779547 19109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:25.779693 19094 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:33:25.779783 19094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/ts-2/wal
uuid: "5cbf45ff2943424594f6a7568d3a93de"
format_stamp: "Formatted at 2026-09-28 06:33:25 on dist-test-slave-rrj5"
server_key: "1e8ae9c7e770cfb5a0b4e6c7c3d46ebd"
server_key_iv: "e9dfa99bb1dcf7c70fd698b27883b499"
server_key_version: "encryptionkey@0"
I20260928 06:33:25.779852 19094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:25.794421 19094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:25.796568 19094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:25.796707 19094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:25.797302 19094 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:33:25.797356 19094 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:25.797396 19094 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:33:25.797423 19094 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 06:33:25 dist-test-slave-rrj5 krb5kdc[18697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577205, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.67@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:33:25.805068 19094 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.67:35149
I20260928 06:33:25.805184 19224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.67:35149 every 8 connection(s)
I20260928 06:33:25.805859 19094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
I20260928 06:33:25.807909 18792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:25.797874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:45445 (local address 127.16.157.126:40157)
0928 06:33:25.797920 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:25.797923 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:25.798196 (+ 273us) server_negotiation.cc:407] Connection header received
0928 06:33:25.798864 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:25.798866 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:25.798883 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:25.798963 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:25.799747 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:25.800492 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:25.801907 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:25.802242 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:25.802332 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:25.804853 (+ 2521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:25.804854 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:25.804855 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:25.804864 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:25.806584 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:25.807008 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:25.807009 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:25.807009 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:25.807061 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:25.807328 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:25.807328 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:25.807329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:25.807431 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:25.807481 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:25.807785 (+ 304us) server_negotiation.cc:299] Negotiation successful
0928 06:33:25.807846 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 06:33:25.808113 19118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:25.797873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40157 (local address 127.16.157.67:45445)
0928 06:33:25.798039 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:25.798085 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:25.798717 (+ 632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:25.799034 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:25.799039 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:25.799374 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:25.799625 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:25.799641 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:25.800624 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:25.800625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:25.801832 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:25.801835 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:25.802334 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:25.802335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:25.802457 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:25.803006 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:25.803023 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:25.804744 (+ 1721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:25.806664 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:25.806669 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:25.806671 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:25.806935 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:25.807123 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:25.807124 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:25.807124 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:25.807265 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:25.807494 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:25.807499 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:25.807653 (+ 154us) client_negotiation.cc:769] Sending connection context
0928 06:33:25.807762 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 06:33:25.807916 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":91,"threads_started":1}
I20260928 06:33:25.808600 19225 heartbeater.cc:347] Connected to a master server at 127.16.157.126:40157
I20260928 06:33:25.808686 19225 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:25.808851 19225 heartbeater.cc:511] Master 127.16.157.126:40157 requested a full tablet report, sending...
I20260928 06:33:25.809186 18741 ts_manager.cc:194] Registered new tserver with Master: 5cbf45ff2943424594f6a7568d3a93de (127.16.157.67:35149)
I20260928 06:33:25.809677 18741 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.157.67@KRBTEST.COM'} at 127.16.157.67:45445
I20260928 06:33:25.813959 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 19094
I20260928 06:33:25.814055 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/ts-2/wal/instance
I20260928 06:33:25.814241 17013 external_mini_cluster.cc:1428] Setting key 34a0c3edcd5ae59f8a9eccede9fe4497
I20260928 06:33:25.815066 17013 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:33:25.815732 18792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:25.809835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:44941 (local address 127.16.157.126:40157)
0928 06:33:25.809895 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:25.809896 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:25.810120 (+ 224us) server_negotiation.cc:407] Connection header received
0928 06:33:25.810164 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:25.810164 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:25.810174 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:25.810233 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:25.810567 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:25.811019 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:25.811892 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:25.812063 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:25.812112 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:25.813022 (+ 910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:25.813024 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:25.813024 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:25.813030 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:25.814598 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:25.814946 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:25.814947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:25.814947 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:25.815011 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:25.815216 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:25.815217 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:25.815218 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:25.815328 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:25.815403 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:25.815565 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 06:33:25.815641 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 06:33:25.815760 19231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:25.809666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40157 (local address 127.16.157.67:44941)
0928 06:33:25.810049 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:25.810059 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:25.810112 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:25.810235 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:25.810236 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:25.810422 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:25.810483 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:25.810485 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:25.811092 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:25.811093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:25.811813 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:25.811815 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:25.812131 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:25.812132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:25.812203 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:25.812588 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:25.812593 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:25.812941 (+ 348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:25.814701 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:25.814702 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:25.814702 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:25.814889 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:25.815069 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:25.815070 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:25.815071 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:25.815135 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:25.815408 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:25.815409 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:25.815452 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 06:33:25.815540 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 06:33:25.815642 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":97,"threads_started":1}
Sep 28 06:33:25 dist-test-slave-rrj5 krb5kdc[18697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 06:33:25 dist-test-slave-rrj5 krb5kdc[18697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:33:25.840986 18792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:25.833409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49210 (local address 127.16.157.126:40157)
0928 06:33:25.833482 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:25.833484 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:25.833496 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:33:25.833524 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:25.833525 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:25.833537 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:25.833627 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:25.834213 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:25.834901 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:25.835655 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:25.835938 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:25.836018 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:25.838505 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:25.838506 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:25.838507 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:25.838514 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:25.839957 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:25.840279 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:25.840280 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:25.840281 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:25.840341 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:25.840514 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:25.840515 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:25.840515 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:25.840640 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:25.840735 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:25.840819 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 06:33:25.840896 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 06:33:25.842868 18741 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49210:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:33:25.844419 18741 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:33:25.852061 19242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:25.846854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.66:41079 (local address 127.0.0.1:39538)
0928 06:33:25.846977 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:25.847019 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:25.847088 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:25.847907 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:25.847908 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:25.847910 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:33:25.847940 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:25.847941 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:25.849287 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:25.849288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:25.850776 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:25.850779 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:25.851730 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:25.851731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:25.851808 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:25.851823 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 06:33:25.851873 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 06:33:25.851943 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"mutex_wait_us":116,"thread_start_us":68,"threads_started":1}
I20260928 06:33:25.852120 19246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:25.847501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53156 (local address 127.16.157.67:35149)
0928 06:33:25.847700 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:25.847705 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:25.847715 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:33:25.847751 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:25.847753 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:25.847768 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:25.847845 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:33:25.848011 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:25.849213 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:25.850877 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:25.851628 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:25.851705 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:25.851766 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:25.851983 (+ 217us) server_negotiation.cc:299] Negotiation successful
0928 06:33:25.852025 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":106,"threads_started":1}
I20260928 06:33:25.852365 19245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:25.847243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55714 (local address 127.16.157.65:46219)
0928 06:33:25.847431 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:25.847436 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:25.847447 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:33:25.847487 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:25.847489 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:25.847503 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:25.847774 (+ 271us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:33:25.847838 (+ 64us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:25.849183 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:25.851046 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:25.851731 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:25.852105 (+ 374us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:25.852216 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:25.852242 (+ 26us) server_negotiation.cc:299] Negotiation successful
0928 06:33:25.852278 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":91,"threads_started":1}
I20260928 06:33:25.852070 19243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:25.846854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.65:46219 (local address 127.0.0.1:55714)
0928 06:33:25.846977 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:25.847019 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:25.847088 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:25.847611 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:25.847616 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:25.847618 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:33:25.847692 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:25.847694 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:25.849283 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:25.849284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:25.850712 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:25.850715 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:25.851830 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:25.851831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:25.851888 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:25.851904 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 06:33:25.851931 (+ 27us) client_negotiation.cc:240] Negotiation successful
0928 06:33:25.851974 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"mutex_wait_us":154,"thread_start_us":80,"threads_started":1}
I20260928 06:33:25.852692 19244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:25.847063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.67:35149 (local address 127.0.0.1:53156)
0928 06:33:25.847170 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:25.847182 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:25.847233 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:25.847858 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:25.847860 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:25.847862 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:33:25.847921 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:25.847923 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:25.849294 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:25.849295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:25.850776 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:25.850779 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:25.851795 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:25.851795 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:25.851842 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:25.851844 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:33:25.851890 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 06:33:25.851943 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"mutex_wait_us":95,"thread_start_us":66,"threads_started":1}
I20260928 06:33:25.853278 19247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:25.847535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39538 (local address 127.16.157.66:41079)
0928 06:33:25.847713 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:25.847717 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:25.847728 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:33:25.847772 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:25.847774 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:25.847786 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:25.848014 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:33:25.848091 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:25.849183 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:25.850879 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:25.851628 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:25.853044 (+ 1416us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:25.853108 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:25.853132 (+ 24us) server_negotiation.cc:299] Negotiation successful
0928 06:33:25.853182 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":74,"threads_started":1}
I20260928 06:33:25.853281 18879 tablet_service.cc:1467] Processing CreateTablet for tablet 33791d82e2ba4bd7921fc3a8f7cd3e0c (DEFAULT_TABLE table=test-table [id=06cb033b04a440ed99fc65269820330a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:33:25.853366 19148 tablet_service.cc:1467] Processing CreateTablet for tablet 33791d82e2ba4bd7921fc3a8f7cd3e0c (DEFAULT_TABLE table=test-table [id=06cb033b04a440ed99fc65269820330a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:33:25.853660 18879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33791d82e2ba4bd7921fc3a8f7cd3e0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:25.853726 19148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33791d82e2ba4bd7921fc3a8f7cd3e0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:25.853880 19019 tablet_service.cc:1467] Processing CreateTablet for tablet 33791d82e2ba4bd7921fc3a8f7cd3e0c (DEFAULT_TABLE table=test-table [id=06cb033b04a440ed99fc65269820330a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:33:25.854189 19019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33791d82e2ba4bd7921fc3a8f7cd3e0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:25.857554 19248 tablet_bootstrap.cc:492] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P 5cbf45ff2943424594f6a7568d3a93de: Bootstrap starting.
I20260928 06:33:25.857995 19250 tablet_bootstrap.cc:492] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P bffb4fdc681a4dd39a816b4a9706a7af: Bootstrap starting.
I20260928 06:33:25.858529 19248 tablet_bootstrap.cc:654] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P 5cbf45ff2943424594f6a7568d3a93de: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:25.858605 19250 tablet_bootstrap.cc:654] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P bffb4fdc681a4dd39a816b4a9706a7af: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:25.858855 19248 log.cc:824] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P 5cbf45ff2943424594f6a7568d3a93de: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:25.858913 19250 log.cc:824] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P bffb4fdc681a4dd39a816b4a9706a7af: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:25.859234 19249 tablet_bootstrap.cc:492] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P 25aa964a114141aeb4b2374eaf7cb85f: Bootstrap starting.
I20260928 06:33:25.859555 19248 tablet_bootstrap.cc:492] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P 5cbf45ff2943424594f6a7568d3a93de: No bootstrap required, opened a new log
I20260928 06:33:25.859638 19248 ts_tablet_manager.cc:1397] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P 5cbf45ff2943424594f6a7568d3a93de: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:33:25.859793 19249 tablet_bootstrap.cc:654] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P 25aa964a114141aeb4b2374eaf7cb85f: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:25.860100 19249 log.cc:824] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P 25aa964a114141aeb4b2374eaf7cb85f: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:25.860836 19250 tablet_bootstrap.cc:492] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P bffb4fdc681a4dd39a816b4a9706a7af: No bootstrap required, opened a new log
I20260928 06:33:25.860900 19250 ts_tablet_manager.cc:1397] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P bffb4fdc681a4dd39a816b4a9706a7af: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 06:33:25.861449 19248 raft_consensus.cc:348] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P 5cbf45ff2943424594f6a7568d3a93de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25aa964a114141aeb4b2374eaf7cb85f" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 46219 } } peers { permanent_uuid: "5cbf45ff2943424594f6a7568d3a93de" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35149 } } peers { permanent_uuid: "bffb4fdc681a4dd39a816b4a9706a7af" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 41079 } }
I20260928 06:33:25.861630 19248 raft_consensus.cc:374] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P 5cbf45ff2943424594f6a7568d3a93de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:25.861677 19248 raft_consensus.cc:729] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P 5cbf45ff2943424594f6a7568d3a93de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cbf45ff2943424594f6a7568d3a93de, State: Initialized, Role: FOLLOWER
I20260928 06:33:25.861790 19248 consensus_queue.cc:260] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P 5cbf45ff2943424594f6a7568d3a93de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25aa964a114141aeb4b2374eaf7cb85f" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 46219 } } peers { permanent_uuid: "5cbf45ff2943424594f6a7568d3a93de" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35149 } } peers { permanent_uuid: "bffb4fdc681a4dd39a816b4a9706a7af" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 41079 } }
I20260928 06:33:25.862079 19225 heartbeater.cc:503] Master 127.16.157.126:40157 was elected leader, sending a full tablet report...
I20260928 06:33:25.862352 19248 ts_tablet_manager.cc:1428] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P 5cbf45ff2943424594f6a7568d3a93de: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260928 06:33:25.863049 19250 raft_consensus.cc:348] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P bffb4fdc681a4dd39a816b4a9706a7af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25aa964a114141aeb4b2374eaf7cb85f" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 46219 } } peers { permanent_uuid: "5cbf45ff2943424594f6a7568d3a93de" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35149 } } peers { permanent_uuid: "bffb4fdc681a4dd39a816b4a9706a7af" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 41079 } }
I20260928 06:33:25.863327 19250 raft_consensus.cc:374] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P bffb4fdc681a4dd39a816b4a9706a7af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:25.863372 19250 raft_consensus.cc:729] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P bffb4fdc681a4dd39a816b4a9706a7af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bffb4fdc681a4dd39a816b4a9706a7af, State: Initialized, Role: FOLLOWER
I20260928 06:33:25.863474 19250 consensus_queue.cc:260] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P bffb4fdc681a4dd39a816b4a9706a7af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25aa964a114141aeb4b2374eaf7cb85f" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 46219 } } peers { permanent_uuid: "5cbf45ff2943424594f6a7568d3a93de" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35149 } } peers { permanent_uuid: "bffb4fdc681a4dd39a816b4a9706a7af" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 41079 } }
I20260928 06:33:25.863711 19250 ts_tablet_manager.cc:1428] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P bffb4fdc681a4dd39a816b4a9706a7af: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:33:25.863754 19085 heartbeater.cc:503] Master 127.16.157.126:40157 was elected leader, sending a full tablet report...
I20260928 06:33:25.864082 19249 tablet_bootstrap.cc:492] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P 25aa964a114141aeb4b2374eaf7cb85f: No bootstrap required, opened a new log
I20260928 06:33:25.864156 19249 ts_tablet_manager.cc:1397] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P 25aa964a114141aeb4b2374eaf7cb85f: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260928 06:33:25.865952 19249 raft_consensus.cc:348] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P 25aa964a114141aeb4b2374eaf7cb85f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25aa964a114141aeb4b2374eaf7cb85f" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 46219 } } peers { permanent_uuid: "5cbf45ff2943424594f6a7568d3a93de" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35149 } } peers { permanent_uuid: "bffb4fdc681a4dd39a816b4a9706a7af" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 41079 } }
I20260928 06:33:25.866101 19249 raft_consensus.cc:374] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P 25aa964a114141aeb4b2374eaf7cb85f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:25.866142 19249 raft_consensus.cc:729] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P 25aa964a114141aeb4b2374eaf7cb85f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25aa964a114141aeb4b2374eaf7cb85f, State: Initialized, Role: FOLLOWER
I20260928 06:33:25.866243 19249 consensus_queue.cc:260] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P 25aa964a114141aeb4b2374eaf7cb85f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25aa964a114141aeb4b2374eaf7cb85f" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 46219 } } peers { permanent_uuid: "5cbf45ff2943424594f6a7568d3a93de" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35149 } } peers { permanent_uuid: "bffb4fdc681a4dd39a816b4a9706a7af" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 41079 } }
I20260928 06:33:25.866479 19249 ts_tablet_manager.cc:1428] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P 25aa964a114141aeb4b2374eaf7cb85f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 06:33:25.866532 18945 heartbeater.cc:503] Master 127.16.157.126:40157 was elected leader, sending a full tablet report...
W20260928 06:33:25.894891 19086 tablet.cc:2369] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P bffb4fdc681a4dd39a816b4a9706a7af: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:33:25.948728 19256 raft_consensus.cc:482] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P 25aa964a114141aeb4b2374eaf7cb85f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:33:25.948869 19256 raft_consensus.cc:504] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P 25aa964a114141aeb4b2374eaf7cb85f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25aa964a114141aeb4b2374eaf7cb85f" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 46219 } } peers { permanent_uuid: "5cbf45ff2943424594f6a7568d3a93de" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35149 } } peers { permanent_uuid: "bffb4fdc681a4dd39a816b4a9706a7af" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 41079 } }
I20260928 06:33:25.949288 19256 leader_election.cc:288] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P 25aa964a114141aeb4b2374eaf7cb85f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5cbf45ff2943424594f6a7568d3a93de (127.16.157.67:35149), bffb4fdc681a4dd39a816b4a9706a7af (127.16.157.66:41079)
I20260928 06:33:25.952874 19247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:25.949645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:43855 (local address 127.16.157.66:41079)
0928 06:33:25.949698 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:25.949702 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:25.949836 (+ 134us) server_negotiation.cc:407] Connection header received
0928 06:33:25.949905 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:25.949906 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:25.949915 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:25.949999 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:33:25.950174 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:25.950932 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:25.952119 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:25.952484 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:25.952570 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:25.952580 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:25.952765 (+ 185us) server_negotiation.cc:299] Negotiation successful
0928 06:33:25.952802 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 06:33:25.952932 19257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:25.949493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.66:41079 (local address 127.16.157.65:43855)
0928 06:33:25.949757 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:25.949767 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:25.949843 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:25.950004 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:25.950005 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:25.950007 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:33:25.950085 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:25.950087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:25.951039 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:25.951040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:25.952038 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:25.952039 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:25.952569 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:25.952570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:25.952681 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:25.952685 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:33:25.952751 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 06:33:25.952793 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":43,"thread_start_us":123,"threads_started":1}
I20260928 06:33:25.953060 19246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:25.949698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:59875 (local address 127.16.157.67:35149)
0928 06:33:25.949742 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:25.949744 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:25.949753 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:33:25.949782 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:25.949783 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:25.949792 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:25.949857 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:33:25.950084 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:25.950593 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:25.951903 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:25.952506 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:25.952577 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:25.952585 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:25.952958 (+ 373us) server_negotiation.cc:299] Negotiation successful
0928 06:33:25.953003 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 06:33:25.953080 18946 tablet.cc:2369] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P 25aa964a114141aeb4b2374eaf7cb85f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:33:25.952932 18838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:25.949493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.67:35149 (local address 127.16.157.65:59875)
0928 06:33:25.949532 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:25.949545 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:25.949603 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:25.949921 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:25.949922 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:25.949924 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:33:25.950014 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:25.950016 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:25.950705 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:25.950706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:25.951810 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:25.951812 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:25.952629 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:25.952630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:25.952747 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:25.952750 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:33:25.952796 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 06:33:25.952839 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260928 06:33:25.953600 19170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33791d82e2ba4bd7921fc3a8f7cd3e0c" candidate_uuid: "25aa964a114141aeb4b2374eaf7cb85f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cbf45ff2943424594f6a7568d3a93de" is_pre_election: true
I20260928 06:33:25.953662 19039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33791d82e2ba4bd7921fc3a8f7cd3e0c" candidate_uuid: "25aa964a114141aeb4b2374eaf7cb85f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bffb4fdc681a4dd39a816b4a9706a7af" is_pre_election: true
I20260928 06:33:25.953774 19039 raft_consensus.cc:2454] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P bffb4fdc681a4dd39a816b4a9706a7af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25aa964a114141aeb4b2374eaf7cb85f in term 0.
I20260928 06:33:25.953771 19170 raft_consensus.cc:2454] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P 5cbf45ff2943424594f6a7568d3a93de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25aa964a114141aeb4b2374eaf7cb85f in term 0.
I20260928 06:33:25.953984 18831 leader_election.cc:302] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P 25aa964a114141aeb4b2374eaf7cb85f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25aa964a114141aeb4b2374eaf7cb85f, bffb4fdc681a4dd39a816b4a9706a7af; no voters:
I20260928 06:33:25.954136 19256 raft_consensus.cc:2781] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P 25aa964a114141aeb4b2374eaf7cb85f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:33:25.954212 19256 raft_consensus.cc:482] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P 25aa964a114141aeb4b2374eaf7cb85f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:33:25.954244 19256 raft_consensus.cc:3037] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P 25aa964a114141aeb4b2374eaf7cb85f [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:25.954954 19256 raft_consensus.cc:504] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P 25aa964a114141aeb4b2374eaf7cb85f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25aa964a114141aeb4b2374eaf7cb85f" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 46219 } } peers { permanent_uuid: "5cbf45ff2943424594f6a7568d3a93de" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35149 } } peers { permanent_uuid: "bffb4fdc681a4dd39a816b4a9706a7af" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 41079 } }
I20260928 06:33:25.955101 19256 leader_election.cc:288] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P 25aa964a114141aeb4b2374eaf7cb85f [CANDIDATE]: Term 1 election: Requested vote from peers 5cbf45ff2943424594f6a7568d3a93de (127.16.157.67:35149), bffb4fdc681a4dd39a816b4a9706a7af (127.16.157.66:41079)
I20260928 06:33:25.955232 19170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33791d82e2ba4bd7921fc3a8f7cd3e0c" candidate_uuid: "25aa964a114141aeb4b2374eaf7cb85f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cbf45ff2943424594f6a7568d3a93de"
I20260928 06:33:25.955250 19039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33791d82e2ba4bd7921fc3a8f7cd3e0c" candidate_uuid: "25aa964a114141aeb4b2374eaf7cb85f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bffb4fdc681a4dd39a816b4a9706a7af"
I20260928 06:33:25.955302 19170 raft_consensus.cc:3037] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P 5cbf45ff2943424594f6a7568d3a93de [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:25.955335 19039 raft_consensus.cc:3037] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P bffb4fdc681a4dd39a816b4a9706a7af [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:25.956192 19170 raft_consensus.cc:2454] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P 5cbf45ff2943424594f6a7568d3a93de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25aa964a114141aeb4b2374eaf7cb85f in term 1.
I20260928 06:33:25.956204 19039 raft_consensus.cc:2454] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P bffb4fdc681a4dd39a816b4a9706a7af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25aa964a114141aeb4b2374eaf7cb85f in term 1.
I20260928 06:33:25.956637 18832 leader_election.cc:302] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P 25aa964a114141aeb4b2374eaf7cb85f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25aa964a114141aeb4b2374eaf7cb85f, 5cbf45ff2943424594f6a7568d3a93de; no voters:
I20260928 06:33:25.956763 19256 raft_consensus.cc:2781] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P 25aa964a114141aeb4b2374eaf7cb85f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:33:25.956934 19256 raft_consensus.cc:686] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P 25aa964a114141aeb4b2374eaf7cb85f [term 1 LEADER]: Becoming Leader. State: Replica: 25aa964a114141aeb4b2374eaf7cb85f, State: Running, Role: LEADER
I20260928 06:33:25.957028 19256 consensus_queue.cc:237] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P 25aa964a114141aeb4b2374eaf7cb85f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25aa964a114141aeb4b2374eaf7cb85f" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 46219 } } peers { permanent_uuid: "5cbf45ff2943424594f6a7568d3a93de" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35149 } } peers { permanent_uuid: "bffb4fdc681a4dd39a816b4a9706a7af" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 41079 } }
I20260928 06:33:25.957911 18742 catalog_manager.cc:5495] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P 25aa964a114141aeb4b2374eaf7cb85f reported cstate change: term changed from 0 to 1, leader changed from <none> to 25aa964a114141aeb4b2374eaf7cb85f (127.16.157.65). New cstate: current_term: 1 leader_uuid: "25aa964a114141aeb4b2374eaf7cb85f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25aa964a114141aeb4b2374eaf7cb85f" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 46219 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5cbf45ff2943424594f6a7568d3a93de" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bffb4fdc681a4dd39a816b4a9706a7af" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 41079 } health_report { overall_health: UNKNOWN } } }
Sep 28 06:33:25 dist-test-slave-rrj5 krb5kdc[18697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577205, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.126@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:33:25.989733 19265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:25.982309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40157 (local address 127.0.0.1:49226)
0928 06:33:25.982463 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:25.982475 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:25.982550 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:25.982721 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:25.982722 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:25.983123 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:25.983179 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:25.983181 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:25.983897 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:25.983898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:25.984673 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:25.984676 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:25.984995 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:25.984995 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:25.985049 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:25.985593 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:25.985599 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:25.987153 (+ 1554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:25.988634 (+ 1481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:25.988641 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:25.988643 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:25.988875 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:25.989058 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:25.989059 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:25.989059 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:25.989122 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:25.989338 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:25.989344 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:25.989391 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 06:33:25.989460 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 06:33:25.989610 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":1,"thread_start_us":93,"threads_started":1}
I20260928 06:33:25.989776 18792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:25.982360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49226 (local address 127.16.157.126:40157)
0928 06:33:25.982444 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:25.982447 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:25.982561 (+ 114us) server_negotiation.cc:407] Connection header received
0928 06:33:25.982638 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:25.982638 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:25.982653 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:25.982695 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:25.983257 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:25.983830 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:25.984753 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:25.984927 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:25.984974 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:25.987237 (+ 2263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:25.987238 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:25.987238 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:25.987244 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:25.988560 (+ 1316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:25.988945 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:25.988946 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:25.988947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:25.988995 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:25.989181 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:25.989181 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:25.989182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:25.989279 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:25.989328 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:25.989485 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 06:33:25.989676 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":122,"server-negotiator.queue_time_us":54}
I20260928 06:33:25.991875 18742 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.16.157.126@KRBTEST.COM'} at 127.0.0.1:49226:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 06:33:25.992081 18742 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:33:25.993496 19019 tablet_service.cc:1467] Processing CreateTablet for tablet c71aae5ea6054892bf7aa2518b290f16 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=42713a2fe42a40dbb183556683e344bb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:33:25.993542 19148 tablet_service.cc:1467] Processing CreateTablet for tablet c71aae5ea6054892bf7aa2518b290f16 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=42713a2fe42a40dbb183556683e344bb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:33:25.993533 18879 tablet_service.cc:1467] Processing CreateTablet for tablet c71aae5ea6054892bf7aa2518b290f16 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=42713a2fe42a40dbb183556683e344bb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:33:25.993701 19148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c71aae5ea6054892bf7aa2518b290f16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:25.993701 18879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c71aae5ea6054892bf7aa2518b290f16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:25.993710 19019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c71aae5ea6054892bf7aa2518b290f16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:25.994829 19250 tablet_bootstrap.cc:492] T c71aae5ea6054892bf7aa2518b290f16 P bffb4fdc681a4dd39a816b4a9706a7af: Bootstrap starting.
I20260928 06:33:25.995095 19249 tablet_bootstrap.cc:492] T c71aae5ea6054892bf7aa2518b290f16 P 25aa964a114141aeb4b2374eaf7cb85f: Bootstrap starting.
I20260928 06:33:25.995525 19249 tablet_bootstrap.cc:654] T c71aae5ea6054892bf7aa2518b290f16 P 25aa964a114141aeb4b2374eaf7cb85f: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:25.995551 19250 tablet_bootstrap.cc:654] T c71aae5ea6054892bf7aa2518b290f16 P bffb4fdc681a4dd39a816b4a9706a7af: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:25.995946 19248 tablet_bootstrap.cc:492] T c71aae5ea6054892bf7aa2518b290f16 P 5cbf45ff2943424594f6a7568d3a93de: Bootstrap starting.
I20260928 06:33:25.996295 19250 tablet_bootstrap.cc:492] T c71aae5ea6054892bf7aa2518b290f16 P bffb4fdc681a4dd39a816b4a9706a7af: No bootstrap required, opened a new log
I20260928 06:33:25.996340 19250 ts_tablet_manager.cc:1397] T c71aae5ea6054892bf7aa2518b290f16 P bffb4fdc681a4dd39a816b4a9706a7af: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:33:25.996521 19248 tablet_bootstrap.cc:654] T c71aae5ea6054892bf7aa2518b290f16 P 5cbf45ff2943424594f6a7568d3a93de: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:25.996490 19250 raft_consensus.cc:348] T c71aae5ea6054892bf7aa2518b290f16 P bffb4fdc681a4dd39a816b4a9706a7af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25aa964a114141aeb4b2374eaf7cb85f" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 46219 } } peers { permanent_uuid: "bffb4fdc681a4dd39a816b4a9706a7af" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 41079 } } peers { permanent_uuid: "5cbf45ff2943424594f6a7568d3a93de" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35149 } }
I20260928 06:33:25.996579 19250 raft_consensus.cc:374] T c71aae5ea6054892bf7aa2518b290f16 P bffb4fdc681a4dd39a816b4a9706a7af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:25.996604 19250 raft_consensus.cc:729] T c71aae5ea6054892bf7aa2518b290f16 P bffb4fdc681a4dd39a816b4a9706a7af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bffb4fdc681a4dd39a816b4a9706a7af, State: Initialized, Role: FOLLOWER
I20260928 06:33:25.996660 19250 consensus_queue.cc:260] T c71aae5ea6054892bf7aa2518b290f16 P bffb4fdc681a4dd39a816b4a9706a7af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25aa964a114141aeb4b2374eaf7cb85f" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 46219 } } peers { permanent_uuid: "bffb4fdc681a4dd39a816b4a9706a7af" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 41079 } } peers { permanent_uuid: "5cbf45ff2943424594f6a7568d3a93de" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35149 } }
I20260928 06:33:25.996773 19250 ts_tablet_manager.cc:1428] T c71aae5ea6054892bf7aa2518b290f16 P bffb4fdc681a4dd39a816b4a9706a7af: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:25.996877 19255 tablet_replica.cc:439] T c71aae5ea6054892bf7aa2518b290f16 P bffb4fdc681a4dd39a816b4a9706a7af: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25aa964a114141aeb4b2374eaf7cb85f" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 46219 } } peers { permanent_uuid: "bffb4fdc681a4dd39a816b4a9706a7af" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 41079 } } peers { permanent_uuid: "5cbf45ff2943424594f6a7568d3a93de" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35149 } } }
I20260928 06:33:25.996986 19255 tablet_replica.cc:442] T c71aae5ea6054892bf7aa2518b290f16 P bffb4fdc681a4dd39a816b4a9706a7af: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:33:25.997184 19249 tablet_bootstrap.cc:492] T c71aae5ea6054892bf7aa2518b290f16 P 25aa964a114141aeb4b2374eaf7cb85f: No bootstrap required, opened a new log
I20260928 06:33:25.997236 19249 ts_tablet_manager.cc:1397] T c71aae5ea6054892bf7aa2518b290f16 P 25aa964a114141aeb4b2374eaf7cb85f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:33:25.997382 19249 raft_consensus.cc:348] T c71aae5ea6054892bf7aa2518b290f16 P 25aa964a114141aeb4b2374eaf7cb85f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25aa964a114141aeb4b2374eaf7cb85f" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 46219 } } peers { permanent_uuid: "bffb4fdc681a4dd39a816b4a9706a7af" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 41079 } } peers { permanent_uuid: "5cbf45ff2943424594f6a7568d3a93de" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35149 } }
I20260928 06:33:25.997449 19249 raft_consensus.cc:374] T c71aae5ea6054892bf7aa2518b290f16 P 25aa964a114141aeb4b2374eaf7cb85f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:25.997468 19249 raft_consensus.cc:729] T c71aae5ea6054892bf7aa2518b290f16 P 25aa964a114141aeb4b2374eaf7cb85f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25aa964a114141aeb4b2374eaf7cb85f, State: Initialized, Role: FOLLOWER
I20260928 06:33:25.997507 19249 consensus_queue.cc:260] T c71aae5ea6054892bf7aa2518b290f16 P 25aa964a114141aeb4b2374eaf7cb85f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25aa964a114141aeb4b2374eaf7cb85f" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 46219 } } peers { permanent_uuid: "bffb4fdc681a4dd39a816b4a9706a7af" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 41079 } } peers { permanent_uuid: "5cbf45ff2943424594f6a7568d3a93de" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35149 } }
I20260928 06:33:25.997653 19249 ts_tablet_manager.cc:1428] T c71aae5ea6054892bf7aa2518b290f16 P 25aa964a114141aeb4b2374eaf7cb85f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:25.997709 19256 tablet_replica.cc:439] T c71aae5ea6054892bf7aa2518b290f16 P 25aa964a114141aeb4b2374eaf7cb85f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25aa964a114141aeb4b2374eaf7cb85f" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 46219 } } peers { permanent_uuid: "bffb4fdc681a4dd39a816b4a9706a7af" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 41079 } } peers { permanent_uuid: "5cbf45ff2943424594f6a7568d3a93de" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35149 } } }
I20260928 06:33:25.997792 19256 tablet_replica.cc:442] T c71aae5ea6054892bf7aa2518b290f16 P 25aa964a114141aeb4b2374eaf7cb85f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:33:25.999127 19248 tablet_bootstrap.cc:492] T c71aae5ea6054892bf7aa2518b290f16 P 5cbf45ff2943424594f6a7568d3a93de: No bootstrap required, opened a new log
I20260928 06:33:25.999177 19248 ts_tablet_manager.cc:1397] T c71aae5ea6054892bf7aa2518b290f16 P 5cbf45ff2943424594f6a7568d3a93de: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.000s
I20260928 06:33:25.999307 19248 raft_consensus.cc:348] T c71aae5ea6054892bf7aa2518b290f16 P 5cbf45ff2943424594f6a7568d3a93de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25aa964a114141aeb4b2374eaf7cb85f" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 46219 } } peers { permanent_uuid: "bffb4fdc681a4dd39a816b4a9706a7af" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 41079 } } peers { permanent_uuid: "5cbf45ff2943424594f6a7568d3a93de" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35149 } }
I20260928 06:33:25.999372 19248 raft_consensus.cc:374] T c71aae5ea6054892bf7aa2518b290f16 P 5cbf45ff2943424594f6a7568d3a93de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:25.999389 19248 raft_consensus.cc:729] T c71aae5ea6054892bf7aa2518b290f16 P 5cbf45ff2943424594f6a7568d3a93de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cbf45ff2943424594f6a7568d3a93de, State: Initialized, Role: FOLLOWER
I20260928 06:33:25.999426 19248 consensus_queue.cc:260] T c71aae5ea6054892bf7aa2518b290f16 P 5cbf45ff2943424594f6a7568d3a93de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25aa964a114141aeb4b2374eaf7cb85f" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 46219 } } peers { permanent_uuid: "bffb4fdc681a4dd39a816b4a9706a7af" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 41079 } } peers { permanent_uuid: "5cbf45ff2943424594f6a7568d3a93de" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35149 } }
I20260928 06:33:25.999511 19248 ts_tablet_manager.cc:1428] T c71aae5ea6054892bf7aa2518b290f16 P 5cbf45ff2943424594f6a7568d3a93de: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:25.999645 19254 tablet_replica.cc:439] T c71aae5ea6054892bf7aa2518b290f16 P 5cbf45ff2943424594f6a7568d3a93de: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25aa964a114141aeb4b2374eaf7cb85f" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 46219 } } peers { permanent_uuid: "bffb4fdc681a4dd39a816b4a9706a7af" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 41079 } } peers { permanent_uuid: "5cbf45ff2943424594f6a7568d3a93de" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35149 } } }
I20260928 06:33:25.999714 19254 tablet_replica.cc:442] T c71aae5ea6054892bf7aa2518b290f16 P 5cbf45ff2943424594f6a7568d3a93de: This TxnStatusTablet replica's current role is: FOLLOWER
W20260928 06:33:26.056661 19226 tablet.cc:2369] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P 5cbf45ff2943424594f6a7568d3a93de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:33:26.181556 19256 raft_consensus.cc:482] T c71aae5ea6054892bf7aa2518b290f16 P 25aa964a114141aeb4b2374eaf7cb85f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:33:26.181699 19256 raft_consensus.cc:504] T c71aae5ea6054892bf7aa2518b290f16 P 25aa964a114141aeb4b2374eaf7cb85f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25aa964a114141aeb4b2374eaf7cb85f" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 46219 } } peers { permanent_uuid: "bffb4fdc681a4dd39a816b4a9706a7af" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 41079 } } peers { permanent_uuid: "5cbf45ff2943424594f6a7568d3a93de" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35149 } }
I20260928 06:33:26.181867 19256 leader_election.cc:288] T c71aae5ea6054892bf7aa2518b290f16 P 25aa964a114141aeb4b2374eaf7cb85f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bffb4fdc681a4dd39a816b4a9706a7af (127.16.157.66:41079), 5cbf45ff2943424594f6a7568d3a93de (127.16.157.67:35149)
I20260928 06:33:26.182101 19170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c71aae5ea6054892bf7aa2518b290f16" candidate_uuid: "25aa964a114141aeb4b2374eaf7cb85f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cbf45ff2943424594f6a7568d3a93de" is_pre_election: true
I20260928 06:33:26.182113 19039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c71aae5ea6054892bf7aa2518b290f16" candidate_uuid: "25aa964a114141aeb4b2374eaf7cb85f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bffb4fdc681a4dd39a816b4a9706a7af" is_pre_election: true
I20260928 06:33:26.182205 19170 raft_consensus.cc:2454] T c71aae5ea6054892bf7aa2518b290f16 P 5cbf45ff2943424594f6a7568d3a93de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25aa964a114141aeb4b2374eaf7cb85f in term 0.
I20260928 06:33:26.182209 19039 raft_consensus.cc:2454] T c71aae5ea6054892bf7aa2518b290f16 P bffb4fdc681a4dd39a816b4a9706a7af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25aa964a114141aeb4b2374eaf7cb85f in term 0.
I20260928 06:33:26.182376 18831 leader_election.cc:302] T c71aae5ea6054892bf7aa2518b290f16 P 25aa964a114141aeb4b2374eaf7cb85f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25aa964a114141aeb4b2374eaf7cb85f, bffb4fdc681a4dd39a816b4a9706a7af; no voters:
I20260928 06:33:26.182494 19256 raft_consensus.cc:2781] T c71aae5ea6054892bf7aa2518b290f16 P 25aa964a114141aeb4b2374eaf7cb85f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:33:26.182544 19256 raft_consensus.cc:482] T c71aae5ea6054892bf7aa2518b290f16 P 25aa964a114141aeb4b2374eaf7cb85f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:33:26.182567 19256 raft_consensus.cc:3037] T c71aae5ea6054892bf7aa2518b290f16 P 25aa964a114141aeb4b2374eaf7cb85f [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:26.183218 19256 raft_consensus.cc:504] T c71aae5ea6054892bf7aa2518b290f16 P 25aa964a114141aeb4b2374eaf7cb85f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25aa964a114141aeb4b2374eaf7cb85f" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 46219 } } peers { permanent_uuid: "bffb4fdc681a4dd39a816b4a9706a7af" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 41079 } } peers { permanent_uuid: "5cbf45ff2943424594f6a7568d3a93de" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35149 } }
I20260928 06:33:26.183373 19256 leader_election.cc:288] T c71aae5ea6054892bf7aa2518b290f16 P 25aa964a114141aeb4b2374eaf7cb85f [CANDIDATE]: Term 1 election: Requested vote from peers bffb4fdc681a4dd39a816b4a9706a7af (127.16.157.66:41079), 5cbf45ff2943424594f6a7568d3a93de (127.16.157.67:35149)
I20260928 06:33:26.183518 19170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c71aae5ea6054892bf7aa2518b290f16" candidate_uuid: "25aa964a114141aeb4b2374eaf7cb85f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cbf45ff2943424594f6a7568d3a93de"
I20260928 06:33:26.183605 19170 raft_consensus.cc:3037] T c71aae5ea6054892bf7aa2518b290f16 P 5cbf45ff2943424594f6a7568d3a93de [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:26.183576 19039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c71aae5ea6054892bf7aa2518b290f16" candidate_uuid: "25aa964a114141aeb4b2374eaf7cb85f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bffb4fdc681a4dd39a816b4a9706a7af"
I20260928 06:33:26.183681 19039 raft_consensus.cc:3037] T c71aae5ea6054892bf7aa2518b290f16 P bffb4fdc681a4dd39a816b4a9706a7af [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:26.184216 19170 raft_consensus.cc:2454] T c71aae5ea6054892bf7aa2518b290f16 P 5cbf45ff2943424594f6a7568d3a93de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25aa964a114141aeb4b2374eaf7cb85f in term 1.
I20260928 06:33:26.184355 19039 raft_consensus.cc:2454] T c71aae5ea6054892bf7aa2518b290f16 P bffb4fdc681a4dd39a816b4a9706a7af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25aa964a114141aeb4b2374eaf7cb85f in term 1.
I20260928 06:33:26.184446 18832 leader_election.cc:302] T c71aae5ea6054892bf7aa2518b290f16 P 25aa964a114141aeb4b2374eaf7cb85f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25aa964a114141aeb4b2374eaf7cb85f, 5cbf45ff2943424594f6a7568d3a93de; no voters:
I20260928 06:33:26.184535 19256 raft_consensus.cc:2781] T c71aae5ea6054892bf7aa2518b290f16 P 25aa964a114141aeb4b2374eaf7cb85f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:33:26.184585 19256 raft_consensus.cc:686] T c71aae5ea6054892bf7aa2518b290f16 P 25aa964a114141aeb4b2374eaf7cb85f [term 1 LEADER]: Becoming Leader. State: Replica: 25aa964a114141aeb4b2374eaf7cb85f, State: Running, Role: LEADER
I20260928 06:33:26.184628 19256 consensus_queue.cc:237] T c71aae5ea6054892bf7aa2518b290f16 P 25aa964a114141aeb4b2374eaf7cb85f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25aa964a114141aeb4b2374eaf7cb85f" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 46219 } } peers { permanent_uuid: "bffb4fdc681a4dd39a816b4a9706a7af" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 41079 } } peers { permanent_uuid: "5cbf45ff2943424594f6a7568d3a93de" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35149 } }
I20260928 06:33:26.184909 19258 tablet_replica.cc:439] T c71aae5ea6054892bf7aa2518b290f16 P 25aa964a114141aeb4b2374eaf7cb85f: TxnStatusTablet state changed. Reason: New leader 25aa964a114141aeb4b2374eaf7cb85f. Latest consensus state: current_term: 1 leader_uuid: "25aa964a114141aeb4b2374eaf7cb85f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25aa964a114141aeb4b2374eaf7cb85f" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 46219 } } peers { permanent_uuid: "bffb4fdc681a4dd39a816b4a9706a7af" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 41079 } } peers { permanent_uuid: "5cbf45ff2943424594f6a7568d3a93de" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35149 } } }
I20260928 06:33:26.184996 19258 tablet_replica.cc:442] T c71aae5ea6054892bf7aa2518b290f16 P 25aa964a114141aeb4b2374eaf7cb85f: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:33:26.185225 19270 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 06:33:26.185333 19270 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 06:33:26.185273 18741 catalog_manager.cc:5495] T c71aae5ea6054892bf7aa2518b290f16 P 25aa964a114141aeb4b2374eaf7cb85f reported cstate change: term changed from 0 to 1, leader changed from <none> to 25aa964a114141aeb4b2374eaf7cb85f (127.16.157.65). New cstate: current_term: 1 leader_uuid: "25aa964a114141aeb4b2374eaf7cb85f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25aa964a114141aeb4b2374eaf7cb85f" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 46219 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bffb4fdc681a4dd39a816b4a9706a7af" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 41079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cbf45ff2943424594f6a7568d3a93de" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35149 } health_report { overall_health: UNKNOWN } } }
I20260928 06:33:26.225113 19265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:26.222530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.65:46219 (local address 127.0.0.1:55724)
0928 06:33:26.222667 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:26.222678 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:26.222731 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:26.222929 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:26.222930 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:26.222931 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:33:26.222980 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:26.222983 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:26.223633 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:26.223634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:26.224203 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:26.224205 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:26.224585 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:26.224586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:26.224649 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:26.224673 (+ 24us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:33:26.224943 (+ 270us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:33:26.224953 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 06:33:26.225004 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 06:33:26.225051 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99}
I20260928 06:33:26.225109 19245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:26.222603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55724 (local address 127.16.157.65:46219)
0928 06:33:26.222648 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:26.222650 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:26.222713 (+ 63us) server_negotiation.cc:407] Connection header received
0928 06:33:26.222815 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:26.222816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:26.222837 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:26.222914 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:33:26.223044 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:26.223544 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:26.224295 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:26.224508 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:26.224585 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:26.224770 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:33:26.224873 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:33:26.224963 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:26.225006 (+ 43us) server_negotiation.cc:299] Negotiation successful
0928 06:33:26.225029 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:33:26.227164 19039 raft_consensus.cc:1261] T c71aae5ea6054892bf7aa2518b290f16 P bffb4fdc681a4dd39a816b4a9706a7af [term 1 FOLLOWER]: Refusing update from remote peer 25aa964a114141aeb4b2374eaf7cb85f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:33:26.227164 19170 raft_consensus.cc:1261] T c71aae5ea6054892bf7aa2518b290f16 P 5cbf45ff2943424594f6a7568d3a93de [term 1 FOLLOWER]: Refusing update from remote peer 25aa964a114141aeb4b2374eaf7cb85f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:33:26.227504 19258 consensus_queue.cc:1035] T c71aae5ea6054892bf7aa2518b290f16 P 25aa964a114141aeb4b2374eaf7cb85f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cbf45ff2943424594f6a7568d3a93de" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:33:26.227622 19256 consensus_queue.cc:1035] T c71aae5ea6054892bf7aa2518b290f16 P 25aa964a114141aeb4b2374eaf7cb85f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bffb4fdc681a4dd39a816b4a9706a7af" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 41079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:33:26.229028 19254 tablet_replica.cc:439] T c71aae5ea6054892bf7aa2518b290f16 P 5cbf45ff2943424594f6a7568d3a93de: TxnStatusTablet state changed. Reason: New leader 25aa964a114141aeb4b2374eaf7cb85f. Latest consensus state: current_term: 1 leader_uuid: "25aa964a114141aeb4b2374eaf7cb85f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25aa964a114141aeb4b2374eaf7cb85f" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 46219 } } peers { permanent_uuid: "bffb4fdc681a4dd39a816b4a9706a7af" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 41079 } } peers { permanent_uuid: "5cbf45ff2943424594f6a7568d3a93de" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35149 } } }
I20260928 06:33:26.229153 19254 tablet_replica.cc:442] T c71aae5ea6054892bf7aa2518b290f16 P 5cbf45ff2943424594f6a7568d3a93de: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:33:26.229243 19255 tablet_replica.cc:439] T c71aae5ea6054892bf7aa2518b290f16 P bffb4fdc681a4dd39a816b4a9706a7af: TxnStatusTablet state changed. Reason: New leader 25aa964a114141aeb4b2374eaf7cb85f. Latest consensus state: current_term: 1 leader_uuid: "25aa964a114141aeb4b2374eaf7cb85f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25aa964a114141aeb4b2374eaf7cb85f" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 46219 } } peers { permanent_uuid: "bffb4fdc681a4dd39a816b4a9706a7af" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 41079 } } peers { permanent_uuid: "5cbf45ff2943424594f6a7568d3a93de" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35149 } } }
I20260928 06:33:26.229332 19255 tablet_replica.cc:442] T c71aae5ea6054892bf7aa2518b290f16 P bffb4fdc681a4dd39a816b4a9706a7af: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:33:26.230674 19255 tablet_replica.cc:439] T c71aae5ea6054892bf7aa2518b290f16 P bffb4fdc681a4dd39a816b4a9706a7af: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "25aa964a114141aeb4b2374eaf7cb85f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25aa964a114141aeb4b2374eaf7cb85f" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 46219 } } peers { permanent_uuid: "bffb4fdc681a4dd39a816b4a9706a7af" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 41079 } } peers { permanent_uuid: "5cbf45ff2943424594f6a7568d3a93de" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35149 } } }
I20260928 06:33:26.230676 19258 tablet_replica.cc:439] T c71aae5ea6054892bf7aa2518b290f16 P 25aa964a114141aeb4b2374eaf7cb85f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "25aa964a114141aeb4b2374eaf7cb85f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25aa964a114141aeb4b2374eaf7cb85f" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 46219 } } peers { permanent_uuid: "bffb4fdc681a4dd39a816b4a9706a7af" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 41079 } } peers { permanent_uuid: "5cbf45ff2943424594f6a7568d3a93de" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35149 } } }
I20260928 06:33:26.230773 19255 tablet_replica.cc:442] T c71aae5ea6054892bf7aa2518b290f16 P bffb4fdc681a4dd39a816b4a9706a7af: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:33:26.230778 19258 tablet_replica.cc:442] T c71aae5ea6054892bf7aa2518b290f16 P 25aa964a114141aeb4b2374eaf7cb85f: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:33:26.230825 19258 tablet_replica.cc:439] T c71aae5ea6054892bf7aa2518b290f16 P 25aa964a114141aeb4b2374eaf7cb85f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "25aa964a114141aeb4b2374eaf7cb85f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25aa964a114141aeb4b2374eaf7cb85f" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 46219 } } peers { permanent_uuid: "bffb4fdc681a4dd39a816b4a9706a7af" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 41079 } } peers { permanent_uuid: "5cbf45ff2943424594f6a7568d3a93de" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35149 } } }
I20260928 06:33:26.230866 19258 tablet_replica.cc:442] T c71aae5ea6054892bf7aa2518b290f16 P 25aa964a114141aeb4b2374eaf7cb85f: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:33:26.231027 19254 tablet_replica.cc:439] T c71aae5ea6054892bf7aa2518b290f16 P 5cbf45ff2943424594f6a7568d3a93de: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "25aa964a114141aeb4b2374eaf7cb85f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25aa964a114141aeb4b2374eaf7cb85f" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 46219 } } peers { permanent_uuid: "bffb4fdc681a4dd39a816b4a9706a7af" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 41079 } } peers { permanent_uuid: "5cbf45ff2943424594f6a7568d3a93de" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35149 } } }
I20260928 06:33:26.231102 19254 tablet_replica.cc:442] T c71aae5ea6054892bf7aa2518b290f16 P 5cbf45ff2943424594f6a7568d3a93de: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:33:26.231369 19275 mvcc.cc:205] Tried to move back new op lower bound from 7334204236703887360 to 7334204236532551680. Current Snapshot: MvccSnapshot[applied={T|T < 7334204236703887360}]
I20260928 06:33:26.231765 19273 mvcc.cc:205] Tried to move back new op lower bound from 7334204236703887360 to 7334204236532551680. Current Snapshot: MvccSnapshot[applied={T|T < 7334204236703887360}]
I20260928 06:33:26.232041 19274 mvcc.cc:205] Tried to move back new op lower bound from 7334204236703887360 to 7334204236532551680. Current Snapshot: MvccSnapshot[applied={T|T < 7334204236703887360}]
I20260928 06:33:26.236332 19245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:26.234078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55732 (local address 127.16.157.65:46219)
0928 06:33:26.234117 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:26.234119 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:26.234160 (+ 41us) server_negotiation.cc:407] Connection header received
0928 06:33:26.234225 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:26.234225 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:26.234233 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:26.234301 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:33:26.234459 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:26.234929 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:26.235651 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:26.235868 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:26.235944 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:26.236102 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:33:26.236133 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:33:26.236188 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:26.236244 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 06:33:26.236262 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:33:26.240912 19245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:26.238583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:35383 (local address 127.16.157.65:46219)
0928 06:33:26.238619 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:26.238621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:26.238777 (+ 156us) server_negotiation.cc:407] Connection header received
0928 06:33:26.238815 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:26.238815 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:26.238823 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:26.238890 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:33:26.239022 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:26.239442 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:26.240209 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:26.240398 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:26.240467 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:26.240649 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:33:26.240676 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:33:26.240748 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:26.240811 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 06:33:26.240827 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:33:26.240996 19284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:26.238548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.65:46219 (local address 127.16.157.65:35383)
0928 06:33:26.238693 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:26.238705 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:26.238768 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:26.238892 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:26.238894 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:26.238895 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:33:26.238950 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:26.238951 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:26.239517 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:26.239518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:26.240124 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:26.240125 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:26.240476 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:26.240476 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:26.240536 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:26.240556 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:33:26.240766 (+ 210us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:33:26.240768 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:33:26.240811 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 06:33:26.240846 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":84,"threads_started":1}
I20260928 06:33:26.243227 19039 raft_consensus.cc:1261] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P bffb4fdc681a4dd39a816b4a9706a7af [term 1 FOLLOWER]: Refusing update from remote peer 25aa964a114141aeb4b2374eaf7cb85f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:33:26.243254 19170 raft_consensus.cc:1261] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P 5cbf45ff2943424594f6a7568d3a93de [term 1 FOLLOWER]: Refusing update from remote peer 25aa964a114141aeb4b2374eaf7cb85f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:33:26.243427 19256 consensus_queue.cc:1035] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P 25aa964a114141aeb4b2374eaf7cb85f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cbf45ff2943424594f6a7568d3a93de" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:33:26.243517 19258 consensus_queue.cc:1035] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P 25aa964a114141aeb4b2374eaf7cb85f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bffb4fdc681a4dd39a816b4a9706a7af" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 41079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:33:26.258081 18742 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49210:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:33:26.258191 18742 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49210:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:33:26.261982 18742 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d9646455aab14be695ee9033c14d8ac0: Sending DeleteTablet for 3 replicas of tablet 33791d82e2ba4bd7921fc3a8f7cd3e0c
I20260928 06:33:26.262320 19019 tablet_service.cc:1514] Processing DeleteTablet for tablet 33791d82e2ba4bd7921fc3a8f7cd3e0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:33:26 UTC) from {username='kudu', principal='kudu/127.16.157.126@KRBTEST.COM'} at 127.0.0.1:39538
I20260928 06:33:26.262497 18742 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49210:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:33:26.262638 18742 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:33:26.262830 19292 tablet_replica.cc:330] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P bffb4fdc681a4dd39a816b4a9706a7af: stopping tablet replica
I20260928 06:33:26.262935 19292 raft_consensus.cc:2229] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P bffb4fdc681a4dd39a816b4a9706a7af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:33:26.262982 19292 raft_consensus.cc:2258] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P bffb4fdc681a4dd39a816b4a9706a7af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:33:26.263250 18879 tablet_service.cc:1514] Processing DeleteTablet for tablet 33791d82e2ba4bd7921fc3a8f7cd3e0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:33:26 UTC) from {username='kudu', principal='kudu/127.16.157.126@KRBTEST.COM'} at 127.0.0.1:55714
I20260928 06:33:26.263695 19292 ts_tablet_manager.cc:1905] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P bffb4fdc681a4dd39a816b4a9706a7af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:33:26.264024 19293 tablet_replica.cc:330] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P 25aa964a114141aeb4b2374eaf7cb85f: stopping tablet replica
I20260928 06:33:26.264122 19293 raft_consensus.cc:2229] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P 25aa964a114141aeb4b2374eaf7cb85f [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:33:26.264227 19293 raft_consensus.cc:2258] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P 25aa964a114141aeb4b2374eaf7cb85f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:33:26.264477 19019 tablet_service.cc:1467] Processing CreateTablet for tablet e41c8d9ce2064a9c9aa3fcfcd6de048d (DEFAULT_TABLE table=test-table [id=ea3d4b001fd44c6c84384f94a2f2124c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:33:26.264474 19148 tablet_service.cc:1514] Processing DeleteTablet for tablet 33791d82e2ba4bd7921fc3a8f7cd3e0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:33:26 UTC) from {username='kudu', principal='kudu/127.16.157.126@KRBTEST.COM'} at 127.0.0.1:53156
I20260928 06:33:26.264698 19147 tablet_service.cc:1467] Processing CreateTablet for tablet e41c8d9ce2064a9c9aa3fcfcd6de048d (DEFAULT_TABLE table=test-table [id=ea3d4b001fd44c6c84384f94a2f2124c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:33:26.264732 19019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e41c8d9ce2064a9c9aa3fcfcd6de048d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:26.264814 19147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e41c8d9ce2064a9c9aa3fcfcd6de048d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:26.266305 19250 tablet_bootstrap.cc:492] T e41c8d9ce2064a9c9aa3fcfcd6de048d P bffb4fdc681a4dd39a816b4a9706a7af: Bootstrap starting.
I20260928 06:33:26.266320 18879 tablet_service.cc:1467] Processing CreateTablet for tablet e41c8d9ce2064a9c9aa3fcfcd6de048d (DEFAULT_TABLE table=test-table [id=ea3d4b001fd44c6c84384f94a2f2124c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:33:26.266415 18879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e41c8d9ce2064a9c9aa3fcfcd6de048d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:26.266690 19250 tablet_bootstrap.cc:654] T e41c8d9ce2064a9c9aa3fcfcd6de048d P bffb4fdc681a4dd39a816b4a9706a7af: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:26.267660 19248 tablet_bootstrap.cc:492] T e41c8d9ce2064a9c9aa3fcfcd6de048d P 5cbf45ff2943424594f6a7568d3a93de: Bootstrap starting.
I20260928 06:33:26.268057 19248 tablet_bootstrap.cc:654] T e41c8d9ce2064a9c9aa3fcfcd6de048d P 5cbf45ff2943424594f6a7568d3a93de: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:26.268417 19249 tablet_bootstrap.cc:492] T e41c8d9ce2064a9c9aa3fcfcd6de048d P 25aa964a114141aeb4b2374eaf7cb85f: Bootstrap starting.
I20260928 06:33:26.268658 19292 ts_tablet_manager.cc:1918] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P bffb4fdc681a4dd39a816b4a9706a7af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:33:26.268766 19292 log.cc:1196] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P bffb4fdc681a4dd39a816b4a9706a7af: Deleting WAL directory at /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/ts-1/wal/wals/33791d82e2ba4bd7921fc3a8f7cd3e0c
I20260928 06:33:26.268786 19249 tablet_bootstrap.cc:654] T e41c8d9ce2064a9c9aa3fcfcd6de048d P 25aa964a114141aeb4b2374eaf7cb85f: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:26.269115 19292 ts_tablet_manager.cc:1939] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P bffb4fdc681a4dd39a816b4a9706a7af: Deleting consensus metadata
I20260928 06:33:26.269678 19294 tablet_replica.cc:330] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P 5cbf45ff2943424594f6a7568d3a93de: stopping tablet replica
I20260928 06:33:26.269758 19293 ts_tablet_manager.cc:1905] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P 25aa964a114141aeb4b2374eaf7cb85f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:33:26.269804 19294 raft_consensus.cc:2229] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P 5cbf45ff2943424594f6a7568d3a93de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:33:26.269872 19294 raft_consensus.cc:2258] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P 5cbf45ff2943424594f6a7568d3a93de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:33:26.270238 18727 catalog_manager.cc:4841] TS bffb4fdc681a4dd39a816b4a9706a7af (127.16.157.66:41079): tablet 33791d82e2ba4bd7921fc3a8f7cd3e0c (table test-table [id=06cb033b04a440ed99fc65269820330a]) successfully deleted
I20260928 06:33:26.270303 19294 ts_tablet_manager.cc:1905] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P 5cbf45ff2943424594f6a7568d3a93de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:33:26.271391 19294 ts_tablet_manager.cc:1918] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P 5cbf45ff2943424594f6a7568d3a93de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:33:26.271488 19294 log.cc:1196] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P 5cbf45ff2943424594f6a7568d3a93de: Deleting WAL directory at /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/ts-2/wal/wals/33791d82e2ba4bd7921fc3a8f7cd3e0c
I20260928 06:33:26.271535 19293 ts_tablet_manager.cc:1918] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P 25aa964a114141aeb4b2374eaf7cb85f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:33:26.271606 19293 log.cc:1196] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P 25aa964a114141aeb4b2374eaf7cb85f: Deleting WAL directory at /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/ts-0/wal/wals/33791d82e2ba4bd7921fc3a8f7cd3e0c
I20260928 06:33:26.271857 19293 ts_tablet_manager.cc:1939] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P 25aa964a114141aeb4b2374eaf7cb85f: Deleting consensus metadata
I20260928 06:33:26.272131 19294 ts_tablet_manager.cc:1939] T 33791d82e2ba4bd7921fc3a8f7cd3e0c P 5cbf45ff2943424594f6a7568d3a93de: Deleting consensus metadata
I20260928 06:33:26.272253 18728 catalog_manager.cc:4841] TS 25aa964a114141aeb4b2374eaf7cb85f (127.16.157.65:46219): tablet 33791d82e2ba4bd7921fc3a8f7cd3e0c (table test-table [id=06cb033b04a440ed99fc65269820330a]) successfully deleted
I20260928 06:33:26.272567 18728 catalog_manager.cc:4841] TS 5cbf45ff2943424594f6a7568d3a93de (127.16.157.67:35149): tablet 33791d82e2ba4bd7921fc3a8f7cd3e0c (table test-table [id=06cb033b04a440ed99fc65269820330a]) successfully deleted
I20260928 06:33:26.272699 19249 tablet_bootstrap.cc:492] T e41c8d9ce2064a9c9aa3fcfcd6de048d P 25aa964a114141aeb4b2374eaf7cb85f: No bootstrap required, opened a new log
I20260928 06:33:26.272758 19249 ts_tablet_manager.cc:1397] T e41c8d9ce2064a9c9aa3fcfcd6de048d P 25aa964a114141aeb4b2374eaf7cb85f: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260928 06:33:26.272856 19249 raft_consensus.cc:348] T e41c8d9ce2064a9c9aa3fcfcd6de048d P 25aa964a114141aeb4b2374eaf7cb85f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25aa964a114141aeb4b2374eaf7cb85f" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 46219 } } peers { permanent_uuid: "bffb4fdc681a4dd39a816b4a9706a7af" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 41079 } } peers { permanent_uuid: "5cbf45ff2943424594f6a7568d3a93de" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35149 } }
I20260928 06:33:26.272933 19249 raft_consensus.cc:374] T e41c8d9ce2064a9c9aa3fcfcd6de048d P 25aa964a114141aeb4b2374eaf7cb85f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:26.272954 19249 raft_consensus.cc:729] T e41c8d9ce2064a9c9aa3fcfcd6de048d P 25aa964a114141aeb4b2374eaf7cb85f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25aa964a114141aeb4b2374eaf7cb85f, State: Initialized, Role: FOLLOWER
I20260928 06:33:26.272997 19249 consensus_queue.cc:260] T e41c8d9ce2064a9c9aa3fcfcd6de048d P 25aa964a114141aeb4b2374eaf7cb85f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25aa964a114141aeb4b2374eaf7cb85f" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 46219 } } peers { permanent_uuid: "bffb4fdc681a4dd39a816b4a9706a7af" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 41079 } } peers { permanent_uuid: "5cbf45ff2943424594f6a7568d3a93de" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35149 } }
I20260928 06:33:26.273096 19249 ts_tablet_manager.cc:1428] T e41c8d9ce2064a9c9aa3fcfcd6de048d P 25aa964a114141aeb4b2374eaf7cb85f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:26.273882 19248 tablet_bootstrap.cc:492] T e41c8d9ce2064a9c9aa3fcfcd6de048d P 5cbf45ff2943424594f6a7568d3a93de: No bootstrap required, opened a new log
I20260928 06:33:26.273937 19248 ts_tablet_manager.cc:1397] T e41c8d9ce2064a9c9aa3fcfcd6de048d P 5cbf45ff2943424594f6a7568d3a93de: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.000s
I20260928 06:33:26.274072 19248 raft_consensus.cc:348] T e41c8d9ce2064a9c9aa3fcfcd6de048d P 5cbf45ff2943424594f6a7568d3a93de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25aa964a114141aeb4b2374eaf7cb85f" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 46219 } } peers { permanent_uuid: "bffb4fdc681a4dd39a816b4a9706a7af" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 41079 } } peers { permanent_uuid: "5cbf45ff2943424594f6a7568d3a93de" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35149 } }
I20260928 06:33:26.274137 19250 tablet_bootstrap.cc:492] T e41c8d9ce2064a9c9aa3fcfcd6de048d P bffb4fdc681a4dd39a816b4a9706a7af: No bootstrap required, opened a new log
I20260928 06:33:26.274142 19248 raft_consensus.cc:374] T e41c8d9ce2064a9c9aa3fcfcd6de048d P 5cbf45ff2943424594f6a7568d3a93de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:26.274178 19248 raft_consensus.cc:729] T e41c8d9ce2064a9c9aa3fcfcd6de048d P 5cbf45ff2943424594f6a7568d3a93de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cbf45ff2943424594f6a7568d3a93de, State: Initialized, Role: FOLLOWER
I20260928 06:33:26.274191 19250 ts_tablet_manager.cc:1397] T e41c8d9ce2064a9c9aa3fcfcd6de048d P bffb4fdc681a4dd39a816b4a9706a7af: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260928 06:33:26.274222 19248 consensus_queue.cc:260] T e41c8d9ce2064a9c9aa3fcfcd6de048d P 5cbf45ff2943424594f6a7568d3a93de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25aa964a114141aeb4b2374eaf7cb85f" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 46219 } } peers { permanent_uuid: "bffb4fdc681a4dd39a816b4a9706a7af" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 41079 } } peers { permanent_uuid: "5cbf45ff2943424594f6a7568d3a93de" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35149 } }
I20260928 06:33:26.274308 19248 ts_tablet_manager.cc:1428] T e41c8d9ce2064a9c9aa3fcfcd6de048d P 5cbf45ff2943424594f6a7568d3a93de: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:26.274323 19250 raft_consensus.cc:348] T e41c8d9ce2064a9c9aa3fcfcd6de048d P bffb4fdc681a4dd39a816b4a9706a7af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25aa964a114141aeb4b2374eaf7cb85f" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 46219 } } peers { permanent_uuid: "bffb4fdc681a4dd39a816b4a9706a7af" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 41079 } } peers { permanent_uuid: "5cbf45ff2943424594f6a7568d3a93de" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35149 } }
I20260928 06:33:26.274384 19250 raft_consensus.cc:374] T e41c8d9ce2064a9c9aa3fcfcd6de048d P bffb4fdc681a4dd39a816b4a9706a7af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:26.274402 19250 raft_consensus.cc:729] T e41c8d9ce2064a9c9aa3fcfcd6de048d P bffb4fdc681a4dd39a816b4a9706a7af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bffb4fdc681a4dd39a816b4a9706a7af, State: Initialized, Role: FOLLOWER
I20260928 06:33:26.274442 19250 consensus_queue.cc:260] T e41c8d9ce2064a9c9aa3fcfcd6de048d P bffb4fdc681a4dd39a816b4a9706a7af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25aa964a114141aeb4b2374eaf7cb85f" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 46219 } } peers { permanent_uuid: "bffb4fdc681a4dd39a816b4a9706a7af" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 41079 } } peers { permanent_uuid: "5cbf45ff2943424594f6a7568d3a93de" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35149 } }
I20260928 06:33:26.274530 19250 ts_tablet_manager.cc:1428] T e41c8d9ce2064a9c9aa3fcfcd6de048d P bffb4fdc681a4dd39a816b4a9706a7af: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:26.307727 19254 raft_consensus.cc:482] T e41c8d9ce2064a9c9aa3fcfcd6de048d P 5cbf45ff2943424594f6a7568d3a93de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:33:26.307835 19254 raft_consensus.cc:504] T e41c8d9ce2064a9c9aa3fcfcd6de048d P 5cbf45ff2943424594f6a7568d3a93de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25aa964a114141aeb4b2374eaf7cb85f" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 46219 } } peers { permanent_uuid: "bffb4fdc681a4dd39a816b4a9706a7af" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 41079 } } peers { permanent_uuid: "5cbf45ff2943424594f6a7568d3a93de" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35149 } }
I20260928 06:33:26.308173 19254 leader_election.cc:288] T e41c8d9ce2064a9c9aa3fcfcd6de048d P 5cbf45ff2943424594f6a7568d3a93de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 25aa964a114141aeb4b2374eaf7cb85f (127.16.157.65:46219), bffb4fdc681a4dd39a816b4a9706a7af (127.16.157.66:41079)
I20260928 06:33:26.312314 19247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:26.308522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:36949 (local address 127.16.157.66:41079)
0928 06:33:26.308567 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:26.308570 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:26.308596 (+ 26us) server_negotiation.cc:407] Connection header received
0928 06:33:26.308624 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:26.308625 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:26.308634 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:26.308684 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:33:26.309097 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:26.309896 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:26.311348 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:26.311900 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:26.311951 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:26.311960 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:26.312195 (+ 235us) server_negotiation.cc:299] Negotiation successful
0928 06:33:26.312239 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 06:33:26.312565 19298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:26.308347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.66:41079 (local address 127.16.157.67:36949)
0928 06:33:26.308457 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:26.308470 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:26.308543 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:26.308921 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:26.308922 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:26.308924 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:33:26.308995 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:26.308998 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:26.310002 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:26.310003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:26.311236 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:26.311239 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:26.312021 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:26.312022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:26.312095 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:26.312099 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:33:26.312442 (+ 343us) client_negotiation.cc:240] Negotiation successful
0928 06:33:26.312492 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":61,"threads_started":1}
I20260928 06:33:26.312811 19038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e41c8d9ce2064a9c9aa3fcfcd6de048d" candidate_uuid: "5cbf45ff2943424594f6a7568d3a93de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bffb4fdc681a4dd39a816b4a9706a7af" is_pre_election: true
I20260928 06:33:26.312908 19038 raft_consensus.cc:2454] T e41c8d9ce2064a9c9aa3fcfcd6de048d P bffb4fdc681a4dd39a816b4a9706a7af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5cbf45ff2943424594f6a7568d3a93de in term 0.
I20260928 06:33:26.312986 19245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:26.308901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:39975 (local address 127.16.157.65:46219)
0928 06:33:26.308947 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:26.308949 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:26.309069 (+ 120us) server_negotiation.cc:407] Connection header received
0928 06:33:26.309168 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:26.309169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:26.309180 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:26.309213 (+ 33us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:33:26.309474 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:26.310273 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:26.311719 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:26.312274 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:26.312321 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:26.312330 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:26.312883 (+ 553us) server_negotiation.cc:299] Negotiation successful
0928 06:33:26.312923 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 06:33:26.313081 19111 leader_election.cc:302] T e41c8d9ce2064a9c9aa3fcfcd6de048d P 5cbf45ff2943424594f6a7568d3a93de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cbf45ff2943424594f6a7568d3a93de, bffb4fdc681a4dd39a816b4a9706a7af; no voters:
I20260928 06:33:26.313217 19254 raft_consensus.cc:2781] T e41c8d9ce2064a9c9aa3fcfcd6de048d P 5cbf45ff2943424594f6a7568d3a93de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:33:26.313261 19254 raft_consensus.cc:482] T e41c8d9ce2064a9c9aa3fcfcd6de048d P 5cbf45ff2943424594f6a7568d3a93de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:33:26.313287 19254 raft_consensus.cc:3037] T e41c8d9ce2064a9c9aa3fcfcd6de048d P 5cbf45ff2943424594f6a7568d3a93de [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:26.313462 19299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:26.308830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.65:46219 (local address 127.16.157.67:39975)
0928 06:33:26.308992 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:26.309003 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:26.309095 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:26.309282 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:26.309283 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:26.309284 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:33:26.309388 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:26.309390 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:26.310400 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:26.310401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:26.311617 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:26.311620 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:26.312591 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:26.312592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:26.312793 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:26.312797 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:33:26.313338 (+ 541us) client_negotiation.cc:240] Negotiation successful
0928 06:33:26.313380 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":47,"threads_started":1}
I20260928 06:33:26.313880 18899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e41c8d9ce2064a9c9aa3fcfcd6de048d" candidate_uuid: "5cbf45ff2943424594f6a7568d3a93de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25aa964a114141aeb4b2374eaf7cb85f" is_pre_election: true
I20260928 06:33:26.314031 18899 raft_consensus.cc:2454] T e41c8d9ce2064a9c9aa3fcfcd6de048d P 25aa964a114141aeb4b2374eaf7cb85f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5cbf45ff2943424594f6a7568d3a93de in term 0.
I20260928 06:33:26.314203 19254 raft_consensus.cc:504] T e41c8d9ce2064a9c9aa3fcfcd6de048d P 5cbf45ff2943424594f6a7568d3a93de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25aa964a114141aeb4b2374eaf7cb85f" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 46219 } } peers { permanent_uuid: "bffb4fdc681a4dd39a816b4a9706a7af" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 41079 } } peers { permanent_uuid: "5cbf45ff2943424594f6a7568d3a93de" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35149 } }
I20260928 06:33:26.314357 19254 leader_election.cc:288] T e41c8d9ce2064a9c9aa3fcfcd6de048d P 5cbf45ff2943424594f6a7568d3a93de [CANDIDATE]: Term 1 election: Requested vote from peers 25aa964a114141aeb4b2374eaf7cb85f (127.16.157.65:46219), bffb4fdc681a4dd39a816b4a9706a7af (127.16.157.66:41079)
I20260928 06:33:26.314610 18899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e41c8d9ce2064a9c9aa3fcfcd6de048d" candidate_uuid: "5cbf45ff2943424594f6a7568d3a93de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25aa964a114141aeb4b2374eaf7cb85f"
I20260928 06:33:26.314709 18899 raft_consensus.cc:3037] T e41c8d9ce2064a9c9aa3fcfcd6de048d P 25aa964a114141aeb4b2374eaf7cb85f [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:26.314798 19038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e41c8d9ce2064a9c9aa3fcfcd6de048d" candidate_uuid: "5cbf45ff2943424594f6a7568d3a93de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bffb4fdc681a4dd39a816b4a9706a7af"
I20260928 06:33:26.314880 19038 raft_consensus.cc:3037] T e41c8d9ce2064a9c9aa3fcfcd6de048d P bffb4fdc681a4dd39a816b4a9706a7af [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:26.315395 18899 raft_consensus.cc:2454] T e41c8d9ce2064a9c9aa3fcfcd6de048d P 25aa964a114141aeb4b2374eaf7cb85f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cbf45ff2943424594f6a7568d3a93de in term 1.
I20260928 06:33:26.315687 19038 raft_consensus.cc:2454] T e41c8d9ce2064a9c9aa3fcfcd6de048d P bffb4fdc681a4dd39a816b4a9706a7af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cbf45ff2943424594f6a7568d3a93de in term 1.
I20260928 06:33:26.315940 19112 leader_election.cc:302] T e41c8d9ce2064a9c9aa3fcfcd6de048d P 5cbf45ff2943424594f6a7568d3a93de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25aa964a114141aeb4b2374eaf7cb85f, 5cbf45ff2943424594f6a7568d3a93de; no voters:
I20260928 06:33:26.316076 19254 raft_consensus.cc:2781] T e41c8d9ce2064a9c9aa3fcfcd6de048d P 5cbf45ff2943424594f6a7568d3a93de [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:33:26.316198 19254 raft_consensus.cc:686] T e41c8d9ce2064a9c9aa3fcfcd6de048d P 5cbf45ff2943424594f6a7568d3a93de [term 1 LEADER]: Becoming Leader. State: Replica: 5cbf45ff2943424594f6a7568d3a93de, State: Running, Role: LEADER
I20260928 06:33:26.316269 19254 consensus_queue.cc:237] T e41c8d9ce2064a9c9aa3fcfcd6de048d P 5cbf45ff2943424594f6a7568d3a93de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25aa964a114141aeb4b2374eaf7cb85f" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 46219 } } peers { permanent_uuid: "bffb4fdc681a4dd39a816b4a9706a7af" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 41079 } } peers { permanent_uuid: "5cbf45ff2943424594f6a7568d3a93de" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35149 } }
I20260928 06:33:26.317965 18742 catalog_manager.cc:5495] T e41c8d9ce2064a9c9aa3fcfcd6de048d P 5cbf45ff2943424594f6a7568d3a93de reported cstate change: term changed from 0 to 1, leader changed from <none> to 5cbf45ff2943424594f6a7568d3a93de (127.16.157.67). New cstate: current_term: 1 leader_uuid: "5cbf45ff2943424594f6a7568d3a93de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25aa964a114141aeb4b2374eaf7cb85f" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 46219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bffb4fdc681a4dd39a816b4a9706a7af" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 41079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cbf45ff2943424594f6a7568d3a93de" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35149 } health_report { overall_health: HEALTHY } } }
I20260928 06:33:26.334925 19246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:26.331555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53166 (local address 127.16.157.67:35149)
0928 06:33:26.331607 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:26.331609 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:26.331618 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:33:26.331654 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:26.331655 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:26.331676 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:26.331758 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:33:26.331979 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:26.332705 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:26.333889 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:26.334195 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:26.334260 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:26.334443 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:33:26.334563 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:33:26.334627 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:26.334835 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 06:33:26.334854 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 06:33:26.340263 19245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:26.337366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:50177 (local address 127.16.157.65:46219)
0928 06:33:26.337410 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:26.337412 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:26.337431 (+ 19us) server_negotiation.cc:407] Connection header received
0928 06:33:26.337454 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:26.337455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:26.337466 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:26.337525 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:33:26.337647 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:26.338361 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:26.339396 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:26.339683 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:26.339727 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:26.339990 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:33:26.340023 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:33:26.340051 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:26.340172 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 06:33:26.340192 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 06:33:26.340497 19304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:26.337171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.65:46219 (local address 127.16.157.67:50177)
0928 06:33:26.337317 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:26.337329 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:26.337385 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:26.337518 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:26.337519 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:26.337521 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:33:26.337592 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:26.337595 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:26.338442 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:26.338443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:26.339289 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:26.339291 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:26.339798 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:26.339798 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:26.339880 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:26.339894 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:33:26.340094 (+ 200us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:33:26.340095 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 06:33:26.340357 (+ 262us) client_negotiation.cc:240] Negotiation successful
0928 06:33:26.340404 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":88,"threads_started":1}
I20260928 06:33:26.342499 19038 raft_consensus.cc:1261] T e41c8d9ce2064a9c9aa3fcfcd6de048d P bffb4fdc681a4dd39a816b4a9706a7af [term 1 FOLLOWER]: Refusing update from remote peer 5cbf45ff2943424594f6a7568d3a93de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:33:26.342602 18899 raft_consensus.cc:1261] T e41c8d9ce2064a9c9aa3fcfcd6de048d P 25aa964a114141aeb4b2374eaf7cb85f [term 1 FOLLOWER]: Refusing update from remote peer 5cbf45ff2943424594f6a7568d3a93de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:33:26.342746 19254 consensus_queue.cc:1035] T e41c8d9ce2064a9c9aa3fcfcd6de048d P 5cbf45ff2943424594f6a7568d3a93de [LEADER]: Connected to new peer: Peer: permanent_uuid: "bffb4fdc681a4dd39a816b4a9706a7af" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 41079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:33:26.342865 19300 consensus_queue.cc:1035] T e41c8d9ce2064a9c9aa3fcfcd6de048d P 5cbf45ff2943424594f6a7568d3a93de [LEADER]: Connected to new peer: Peer: permanent_uuid: "25aa964a114141aeb4b2374eaf7cb85f" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 46219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:33:26.351334 19284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:26.348229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.67:35149 (local address 127.16.157.65:47649)
0928 06:33:26.348271 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:26.348282 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:26.348321 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:26.348536 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:26.348537 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:26.348538 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:33:26.348580 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:26.348581 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:26.349491 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:26.349492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:26.350406 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:26.350409 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:26.350879 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:26.350880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:26.350942 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:26.350950 (+ 8us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:33:26.351171 (+ 221us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:33:26.351174 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:33:26.351215 (+ 41us) client_negotiation.cc:240] Negotiation successful
0928 06:33:26.351262 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260928 06:33:26.351588 19246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:26.348315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:47649 (local address 127.16.157.67:35149)
0928 06:33:26.348382 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:26.348384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:26.348392 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:33:26.348411 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:26.348411 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:26.348420 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:26.348473 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:33:26.348660 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:26.349397 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:26.350499 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:26.350796 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:26.350866 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:26.351050 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:33:26.351083 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:33:26.351137 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:26.351502 (+ 365us) server_negotiation.cc:299] Negotiation successful
0928 06:33:26.351523 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260928 06:33:26.351822 19275 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 06:33:27.876658 18740 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49210:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:33:27.876778 18740 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49210:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:33:27.877714 18740 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d9646455aab14be695ee9033c14d8ac0: Sending DeleteTablet for 3 replicas of tablet e41c8d9ce2064a9c9aa3fcfcd6de048d
I20260928 06:33:27.878019 19019 tablet_service.cc:1514] Processing DeleteTablet for tablet e41c8d9ce2064a9c9aa3fcfcd6de048d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:33:27 UTC) from {username='kudu', principal='kudu/127.16.157.126@KRBTEST.COM'} at 127.0.0.1:39538
I20260928 06:33:27.878222 19330 tablet_replica.cc:330] T e41c8d9ce2064a9c9aa3fcfcd6de048d P bffb4fdc681a4dd39a816b4a9706a7af: stopping tablet replica
I20260928 06:33:27.878209 18878 tablet_service.cc:1514] Processing DeleteTablet for tablet e41c8d9ce2064a9c9aa3fcfcd6de048d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:33:27 UTC) from {username='kudu', principal='kudu/127.16.157.126@KRBTEST.COM'} at 127.0.0.1:55714
I20260928 06:33:27.878278 19330 raft_consensus.cc:2229] T e41c8d9ce2064a9c9aa3fcfcd6de048d P bffb4fdc681a4dd39a816b4a9706a7af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:33:27.878319 19330 raft_consensus.cc:2258] T e41c8d9ce2064a9c9aa3fcfcd6de048d P bffb4fdc681a4dd39a816b4a9706a7af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:33:27.878446 19331 tablet_replica.cc:330] T e41c8d9ce2064a9c9aa3fcfcd6de048d P 25aa964a114141aeb4b2374eaf7cb85f: stopping tablet replica
I20260928 06:33:27.878508 19331 raft_consensus.cc:2229] T e41c8d9ce2064a9c9aa3fcfcd6de048d P 25aa964a114141aeb4b2374eaf7cb85f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:33:27.878546 19330 ts_tablet_manager.cc:1905] T e41c8d9ce2064a9c9aa3fcfcd6de048d P bffb4fdc681a4dd39a816b4a9706a7af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:33:27.878579 19331 raft_consensus.cc:2258] T e41c8d9ce2064a9c9aa3fcfcd6de048d P 25aa964a114141aeb4b2374eaf7cb85f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:33:27.878710 19147 tablet_service.cc:1514] Processing DeleteTablet for tablet e41c8d9ce2064a9c9aa3fcfcd6de048d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:33:27 UTC) from {username='kudu', principal='kudu/127.16.157.126@KRBTEST.COM'} at 127.0.0.1:53156
I20260928 06:33:27.878836 19331 ts_tablet_manager.cc:1905] T e41c8d9ce2064a9c9aa3fcfcd6de048d P 25aa964a114141aeb4b2374eaf7cb85f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:33:27.878955 19332 tablet_replica.cc:330] T e41c8d9ce2064a9c9aa3fcfcd6de048d P 5cbf45ff2943424594f6a7568d3a93de: stopping tablet replica
I20260928 06:33:27.879033 19332 raft_consensus.cc:2229] T e41c8d9ce2064a9c9aa3fcfcd6de048d P 5cbf45ff2943424594f6a7568d3a93de [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:33:27.879170 19332 raft_consensus.cc:2258] T e41c8d9ce2064a9c9aa3fcfcd6de048d P 5cbf45ff2943424594f6a7568d3a93de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:33:27.879462 19332 ts_tablet_manager.cc:1905] T e41c8d9ce2064a9c9aa3fcfcd6de048d P 5cbf45ff2943424594f6a7568d3a93de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:33:27.880373 19330 ts_tablet_manager.cc:1918] T e41c8d9ce2064a9c9aa3fcfcd6de048d P bffb4fdc681a4dd39a816b4a9706a7af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 06:33:27.880457 19330 log.cc:1196] T e41c8d9ce2064a9c9aa3fcfcd6de048d P bffb4fdc681a4dd39a816b4a9706a7af: Deleting WAL directory at /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/ts-1/wal/wals/e41c8d9ce2064a9c9aa3fcfcd6de048d
I20260928 06:33:27.880658 19331 ts_tablet_manager.cc:1918] T e41c8d9ce2064a9c9aa3fcfcd6de048d P 25aa964a114141aeb4b2374eaf7cb85f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 06:33:27.880722 19330 ts_tablet_manager.cc:1939] T e41c8d9ce2064a9c9aa3fcfcd6de048d P bffb4fdc681a4dd39a816b4a9706a7af: Deleting consensus metadata
I20260928 06:33:27.880759 19331 log.cc:1196] T e41c8d9ce2064a9c9aa3fcfcd6de048d P 25aa964a114141aeb4b2374eaf7cb85f: Deleting WAL directory at /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/ts-0/wal/wals/e41c8d9ce2064a9c9aa3fcfcd6de048d
I20260928 06:33:27.881037 19331 ts_tablet_manager.cc:1939] T e41c8d9ce2064a9c9aa3fcfcd6de048d P 25aa964a114141aeb4b2374eaf7cb85f: Deleting consensus metadata
I20260928 06:33:27.881069 18727 catalog_manager.cc:4841] TS bffb4fdc681a4dd39a816b4a9706a7af (127.16.157.66:41079): tablet e41c8d9ce2064a9c9aa3fcfcd6de048d (table test-table [id=ea3d4b001fd44c6c84384f94a2f2124c]) successfully deleted
I20260928 06:33:27.881403 18728 catalog_manager.cc:4841] TS 25aa964a114141aeb4b2374eaf7cb85f (127.16.157.65:46219): tablet e41c8d9ce2064a9c9aa3fcfcd6de048d (table test-table [id=ea3d4b001fd44c6c84384f94a2f2124c]) successfully deleted
I20260928 06:33:27.881469 19332 ts_tablet_manager.cc:1918] T e41c8d9ce2064a9c9aa3fcfcd6de048d P 5cbf45ff2943424594f6a7568d3a93de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 06:33:27.881590 19332 log.cc:1196] T e41c8d9ce2064a9c9aa3fcfcd6de048d P 5cbf45ff2943424594f6a7568d3a93de: Deleting WAL directory at /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/ts-2/wal/wals/e41c8d9ce2064a9c9aa3fcfcd6de048d
I20260928 06:33:27.881909 19332 ts_tablet_manager.cc:1939] T e41c8d9ce2064a9c9aa3fcfcd6de048d P 5cbf45ff2943424594f6a7568d3a93de: Deleting consensus metadata
I20260928 06:33:27.882225 18728 catalog_manager.cc:4841] TS 5cbf45ff2943424594f6a7568d3a93de (127.16.157.67:35149): tablet e41c8d9ce2064a9c9aa3fcfcd6de048d (table test-table [id=ea3d4b001fd44c6c84384f94a2f2124c]) successfully deleted
Sep 28 06:33:27 dist-test-slave-rrj5 krb5kdc[18697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577207, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 06:33:27 dist-test-slave-rrj5 krb5kdc[18697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577207, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:33:27.917694 19340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:27.904670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49228 (local address 127.16.157.126:40157)
0928 06:33:27.904824 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:27.904827 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:27.904907 (+ 80us) server_negotiation.cc:407] Connection header received
0928 06:33:27.904992 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:27.904993 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:27.905005 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:27.905072 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:27.905815 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:27.906594 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:27.907544 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:27.907831 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:27.907947 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:27.914948 (+ 7001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:27.914949 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:27.914950 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:27.914958 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:27.916481 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:27.916869 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:27.916870 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:27.916870 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:27.916923 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:27.917116 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:27.917116 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:27.917116 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:27.917232 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:27.917305 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:27.917459 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 06:33:27.917539 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":88,"threads_started":1}
I20260928 06:33:27.919329 18740 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49228:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:33:27.919507 18740 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:33:27.920832 18878 tablet_service.cc:1467] Processing CreateTablet for tablet c172b277ffd44289b788c957ade3b3cd (DEFAULT_TABLE table=test-table [id=2b97f96a36c04dc39e14c904be5a6bb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:33:27.920871 19019 tablet_service.cc:1467] Processing CreateTablet for tablet c172b277ffd44289b788c957ade3b3cd (DEFAULT_TABLE table=test-table [id=2b97f96a36c04dc39e14c904be5a6bb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:33:27.920871 19147 tablet_service.cc:1467] Processing CreateTablet for tablet c172b277ffd44289b788c957ade3b3cd (DEFAULT_TABLE table=test-table [id=2b97f96a36c04dc39e14c904be5a6bb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:33:27.920984 18878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c172b277ffd44289b788c957ade3b3cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:27.921000 19019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c172b277ffd44289b788c957ade3b3cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:27.921000 19147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c172b277ffd44289b788c957ade3b3cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:27.922425 19341 tablet_bootstrap.cc:492] T c172b277ffd44289b788c957ade3b3cd P 5cbf45ff2943424594f6a7568d3a93de: Bootstrap starting.
I20260928 06:33:27.922708 19343 tablet_bootstrap.cc:492] T c172b277ffd44289b788c957ade3b3cd P 25aa964a114141aeb4b2374eaf7cb85f: Bootstrap starting.
I20260928 06:33:27.922799 19341 tablet_bootstrap.cc:654] T c172b277ffd44289b788c957ade3b3cd P 5cbf45ff2943424594f6a7568d3a93de: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:27.923118 19343 tablet_bootstrap.cc:654] T c172b277ffd44289b788c957ade3b3cd P 25aa964a114141aeb4b2374eaf7cb85f: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:27.923177 19342 tablet_bootstrap.cc:492] T c172b277ffd44289b788c957ade3b3cd P bffb4fdc681a4dd39a816b4a9706a7af: Bootstrap starting.
I20260928 06:33:27.923447 19341 tablet_bootstrap.cc:492] T c172b277ffd44289b788c957ade3b3cd P 5cbf45ff2943424594f6a7568d3a93de: No bootstrap required, opened a new log
I20260928 06:33:27.923548 19341 ts_tablet_manager.cc:1397] T c172b277ffd44289b788c957ade3b3cd P 5cbf45ff2943424594f6a7568d3a93de: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:27.923583 19342 tablet_bootstrap.cc:654] T c172b277ffd44289b788c957ade3b3cd P bffb4fdc681a4dd39a816b4a9706a7af: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:27.923700 19341 raft_consensus.cc:348] T c172b277ffd44289b788c957ade3b3cd P 5cbf45ff2943424594f6a7568d3a93de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25aa964a114141aeb4b2374eaf7cb85f" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 46219 } } peers { permanent_uuid: "bffb4fdc681a4dd39a816b4a9706a7af" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 41079 } } peers { permanent_uuid: "5cbf45ff2943424594f6a7568d3a93de" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35149 } }
I20260928 06:33:27.923794 19341 raft_consensus.cc:374] T c172b277ffd44289b788c957ade3b3cd P 5cbf45ff2943424594f6a7568d3a93de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:27.923822 19341 raft_consensus.cc:729] T c172b277ffd44289b788c957ade3b3cd P 5cbf45ff2943424594f6a7568d3a93de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cbf45ff2943424594f6a7568d3a93de, State: Initialized, Role: FOLLOWER
I20260928 06:33:27.923861 19343 tablet_bootstrap.cc:492] T c172b277ffd44289b788c957ade3b3cd P 25aa964a114141aeb4b2374eaf7cb85f: No bootstrap required, opened a new log
I20260928 06:33:27.923940 19343 ts_tablet_manager.cc:1397] T c172b277ffd44289b788c957ade3b3cd P 25aa964a114141aeb4b2374eaf7cb85f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:27.923916 19341 consensus_queue.cc:260] T c172b277ffd44289b788c957ade3b3cd P 5cbf45ff2943424594f6a7568d3a93de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25aa964a114141aeb4b2374eaf7cb85f" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 46219 } } peers { permanent_uuid: "bffb4fdc681a4dd39a816b4a9706a7af" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 41079 } } peers { permanent_uuid: "5cbf45ff2943424594f6a7568d3a93de" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35149 } }
I20260928 06:33:27.924054 19343 raft_consensus.cc:348] T c172b277ffd44289b788c957ade3b3cd P 25aa964a114141aeb4b2374eaf7cb85f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25aa964a114141aeb4b2374eaf7cb85f" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 46219 } } peers { permanent_uuid: "bffb4fdc681a4dd39a816b4a9706a7af" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 41079 } } peers { permanent_uuid: "5cbf45ff2943424594f6a7568d3a93de" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35149 } }
I20260928 06:33:27.924147 19343 raft_consensus.cc:374] T c172b277ffd44289b788c957ade3b3cd P 25aa964a114141aeb4b2374eaf7cb85f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:27.924146 19341 ts_tablet_manager.cc:1428] T c172b277ffd44289b788c957ade3b3cd P 5cbf45ff2943424594f6a7568d3a93de: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 06:33:27.924175 19343 raft_consensus.cc:729] T c172b277ffd44289b788c957ade3b3cd P 25aa964a114141aeb4b2374eaf7cb85f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25aa964a114141aeb4b2374eaf7cb85f, State: Initialized, Role: FOLLOWER
I20260928 06:33:27.924220 19343 consensus_queue.cc:260] T c172b277ffd44289b788c957ade3b3cd P 25aa964a114141aeb4b2374eaf7cb85f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25aa964a114141aeb4b2374eaf7cb85f" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 46219 } } peers { permanent_uuid: "bffb4fdc681a4dd39a816b4a9706a7af" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 41079 } } peers { permanent_uuid: "5cbf45ff2943424594f6a7568d3a93de" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35149 } }
I20260928 06:33:27.924324 19343 ts_tablet_manager.cc:1428] T c172b277ffd44289b788c957ade3b3cd P 25aa964a114141aeb4b2374eaf7cb85f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:27.924810 19342 tablet_bootstrap.cc:492] T c172b277ffd44289b788c957ade3b3cd P bffb4fdc681a4dd39a816b4a9706a7af: No bootstrap required, opened a new log
I20260928 06:33:27.924851 19342 ts_tablet_manager.cc:1397] T c172b277ffd44289b788c957ade3b3cd P bffb4fdc681a4dd39a816b4a9706a7af: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:33:27.924994 19342 raft_consensus.cc:348] T c172b277ffd44289b788c957ade3b3cd P bffb4fdc681a4dd39a816b4a9706a7af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25aa964a114141aeb4b2374eaf7cb85f" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 46219 } } peers { permanent_uuid: "bffb4fdc681a4dd39a816b4a9706a7af" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 41079 } } peers { permanent_uuid: "5cbf45ff2943424594f6a7568d3a93de" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35149 } }
I20260928 06:33:27.925057 19342 raft_consensus.cc:374] T c172b277ffd44289b788c957ade3b3cd P bffb4fdc681a4dd39a816b4a9706a7af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:27.925079 19342 raft_consensus.cc:729] T c172b277ffd44289b788c957ade3b3cd P bffb4fdc681a4dd39a816b4a9706a7af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bffb4fdc681a4dd39a816b4a9706a7af, State: Initialized, Role: FOLLOWER
I20260928 06:33:27.925151 19342 consensus_queue.cc:260] T c172b277ffd44289b788c957ade3b3cd P bffb4fdc681a4dd39a816b4a9706a7af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25aa964a114141aeb4b2374eaf7cb85f" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 46219 } } peers { permanent_uuid: "bffb4fdc681a4dd39a816b4a9706a7af" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 41079 } } peers { permanent_uuid: "5cbf45ff2943424594f6a7568d3a93de" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35149 } }
I20260928 06:33:27.925367 19342 ts_tablet_manager.cc:1428] T c172b277ffd44289b788c957ade3b3cd P bffb4fdc681a4dd39a816b4a9706a7af: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 06:33:27.962709 19316 raft_consensus.cc:482] T c172b277ffd44289b788c957ade3b3cd P 25aa964a114141aeb4b2374eaf7cb85f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:33:27.962829 19316 raft_consensus.cc:504] T c172b277ffd44289b788c957ade3b3cd P 25aa964a114141aeb4b2374eaf7cb85f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25aa964a114141aeb4b2374eaf7cb85f" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 46219 } } peers { permanent_uuid: "bffb4fdc681a4dd39a816b4a9706a7af" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 41079 } } peers { permanent_uuid: "5cbf45ff2943424594f6a7568d3a93de" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35149 } }
I20260928 06:33:27.963028 19316 leader_election.cc:288] T c172b277ffd44289b788c957ade3b3cd P 25aa964a114141aeb4b2374eaf7cb85f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bffb4fdc681a4dd39a816b4a9706a7af (127.16.157.66:41079), 5cbf45ff2943424594f6a7568d3a93de (127.16.157.67:35149)
I20260928 06:33:27.963249 19038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c172b277ffd44289b788c957ade3b3cd" candidate_uuid: "25aa964a114141aeb4b2374eaf7cb85f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bffb4fdc681a4dd39a816b4a9706a7af" is_pre_election: true
I20260928 06:33:27.963271 19170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c172b277ffd44289b788c957ade3b3cd" candidate_uuid: "25aa964a114141aeb4b2374eaf7cb85f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cbf45ff2943424594f6a7568d3a93de" is_pre_election: true
I20260928 06:33:27.963362 19038 raft_consensus.cc:2454] T c172b277ffd44289b788c957ade3b3cd P bffb4fdc681a4dd39a816b4a9706a7af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25aa964a114141aeb4b2374eaf7cb85f in term 0.
I20260928 06:33:27.963383 19170 raft_consensus.cc:2454] T c172b277ffd44289b788c957ade3b3cd P 5cbf45ff2943424594f6a7568d3a93de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25aa964a114141aeb4b2374eaf7cb85f in term 0.
I20260928 06:33:27.963536 18831 leader_election.cc:302] T c172b277ffd44289b788c957ade3b3cd P 25aa964a114141aeb4b2374eaf7cb85f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25aa964a114141aeb4b2374eaf7cb85f, bffb4fdc681a4dd39a816b4a9706a7af; no voters:
I20260928 06:33:27.963665 19316 raft_consensus.cc:2781] T c172b277ffd44289b788c957ade3b3cd P 25aa964a114141aeb4b2374eaf7cb85f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:33:27.963749 19316 raft_consensus.cc:482] T c172b277ffd44289b788c957ade3b3cd P 25aa964a114141aeb4b2374eaf7cb85f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:33:27.963781 19316 raft_consensus.cc:3037] T c172b277ffd44289b788c957ade3b3cd P 25aa964a114141aeb4b2374eaf7cb85f [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:27.966271 19316 raft_consensus.cc:504] T c172b277ffd44289b788c957ade3b3cd P 25aa964a114141aeb4b2374eaf7cb85f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25aa964a114141aeb4b2374eaf7cb85f" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 46219 } } peers { permanent_uuid: "bffb4fdc681a4dd39a816b4a9706a7af" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 41079 } } peers { permanent_uuid: "5cbf45ff2943424594f6a7568d3a93de" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35149 } }
I20260928 06:33:27.966475 19316 leader_election.cc:288] T c172b277ffd44289b788c957ade3b3cd P 25aa964a114141aeb4b2374eaf7cb85f [CANDIDATE]: Term 1 election: Requested vote from peers bffb4fdc681a4dd39a816b4a9706a7af (127.16.157.66:41079), 5cbf45ff2943424594f6a7568d3a93de (127.16.157.67:35149)
I20260928 06:33:27.966603 19038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c172b277ffd44289b788c957ade3b3cd" candidate_uuid: "25aa964a114141aeb4b2374eaf7cb85f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bffb4fdc681a4dd39a816b4a9706a7af"
I20260928 06:33:27.966603 19170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c172b277ffd44289b788c957ade3b3cd" candidate_uuid: "25aa964a114141aeb4b2374eaf7cb85f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cbf45ff2943424594f6a7568d3a93de"
I20260928 06:33:27.966688 19038 raft_consensus.cc:3037] T c172b277ffd44289b788c957ade3b3cd P bffb4fdc681a4dd39a816b4a9706a7af [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:27.966688 19170 raft_consensus.cc:3037] T c172b277ffd44289b788c957ade3b3cd P 5cbf45ff2943424594f6a7568d3a93de [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:27.968158 19038 raft_consensus.cc:2454] T c172b277ffd44289b788c957ade3b3cd P bffb4fdc681a4dd39a816b4a9706a7af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25aa964a114141aeb4b2374eaf7cb85f in term 1.
I20260928 06:33:27.968159 19170 raft_consensus.cc:2454] T c172b277ffd44289b788c957ade3b3cd P 5cbf45ff2943424594f6a7568d3a93de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25aa964a114141aeb4b2374eaf7cb85f in term 1.
I20260928 06:33:27.968343 18831 leader_election.cc:302] T c172b277ffd44289b788c957ade3b3cd P 25aa964a114141aeb4b2374eaf7cb85f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25aa964a114141aeb4b2374eaf7cb85f, bffb4fdc681a4dd39a816b4a9706a7af; no voters:
I20260928 06:33:27.968508 19316 raft_consensus.cc:2781] T c172b277ffd44289b788c957ade3b3cd P 25aa964a114141aeb4b2374eaf7cb85f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:33:27.968595 19316 raft_consensus.cc:686] T c172b277ffd44289b788c957ade3b3cd P 25aa964a114141aeb4b2374eaf7cb85f [term 1 LEADER]: Becoming Leader. State: Replica: 25aa964a114141aeb4b2374eaf7cb85f, State: Running, Role: LEADER
I20260928 06:33:27.968647 19316 consensus_queue.cc:237] T c172b277ffd44289b788c957ade3b3cd P 25aa964a114141aeb4b2374eaf7cb85f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25aa964a114141aeb4b2374eaf7cb85f" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 46219 } } peers { permanent_uuid: "bffb4fdc681a4dd39a816b4a9706a7af" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 41079 } } peers { permanent_uuid: "5cbf45ff2943424594f6a7568d3a93de" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35149 } }
I20260928 06:33:27.969280 18740 catalog_manager.cc:5495] T c172b277ffd44289b788c957ade3b3cd P 25aa964a114141aeb4b2374eaf7cb85f reported cstate change: term changed from 0 to 1, leader changed from <none> to 25aa964a114141aeb4b2374eaf7cb85f (127.16.157.65). New cstate: current_term: 1 leader_uuid: "25aa964a114141aeb4b2374eaf7cb85f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25aa964a114141aeb4b2374eaf7cb85f" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 46219 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bffb4fdc681a4dd39a816b4a9706a7af" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 41079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cbf45ff2943424594f6a7568d3a93de" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35149 } health_report { overall_health: UNKNOWN } } }
I20260928 06:33:27.986316 19350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:27.983530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55746 (local address 127.16.157.65:46219)
0928 06:33:27.983677 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:27.983679 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:27.983687 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:33:27.983717 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:27.983718 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:27.983728 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:27.983786 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:33:27.983955 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:27.984429 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:27.985529 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:27.985807 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:27.985864 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:27.986047 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:33:27.986075 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:33:27.986114 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:27.986206 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 06:33:27.986231 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":99,"threads_started":1}
I20260928 06:33:27.988566 19170 raft_consensus.cc:1261] T c172b277ffd44289b788c957ade3b3cd P 5cbf45ff2943424594f6a7568d3a93de [term 1 FOLLOWER]: Refusing update from remote peer 25aa964a114141aeb4b2374eaf7cb85f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:33:27.988565 19038 raft_consensus.cc:1261] T c172b277ffd44289b788c957ade3b3cd P bffb4fdc681a4dd39a816b4a9706a7af [term 1 FOLLOWER]: Refusing update from remote peer 25aa964a114141aeb4b2374eaf7cb85f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:33:27.988821 19316 consensus_queue.cc:1035] T c172b277ffd44289b788c957ade3b3cd P 25aa964a114141aeb4b2374eaf7cb85f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cbf45ff2943424594f6a7568d3a93de" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:33:27.989025 19316 consensus_queue.cc:1035] T c172b277ffd44289b788c957ade3b3cd P 25aa964a114141aeb4b2374eaf7cb85f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bffb4fdc681a4dd39a816b4a9706a7af" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 41079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:33:27.999454 18741 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49228:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:33:27.999562 18741 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49228:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:33:28.000157 18741 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d9646455aab14be695ee9033c14d8ac0: Sending DeleteTablet for 3 replicas of tablet c172b277ffd44289b788c957ade3b3cd
I20260928 06:33:28.000402 19019 tablet_service.cc:1514] Processing DeleteTablet for tablet c172b277ffd44289b788c957ade3b3cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:33:27 UTC) from {username='kudu', principal='kudu/127.16.157.126@KRBTEST.COM'} at 127.0.0.1:39538
I20260928 06:33:28.000520 19330 tablet_replica.cc:330] T c172b277ffd44289b788c957ade3b3cd P bffb4fdc681a4dd39a816b4a9706a7af: stopping tablet replica
I20260928 06:33:28.000568 19330 raft_consensus.cc:2229] T c172b277ffd44289b788c957ade3b3cd P bffb4fdc681a4dd39a816b4a9706a7af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:33:28.000631 19330 raft_consensus.cc:2258] T c172b277ffd44289b788c957ade3b3cd P bffb4fdc681a4dd39a816b4a9706a7af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:33:28.000566 18741 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49228:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:33:28.000710 18741 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:33:28.000957 19330 ts_tablet_manager.cc:1905] T c172b277ffd44289b788c957ade3b3cd P bffb4fdc681a4dd39a816b4a9706a7af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:33:28.000981 19147 tablet_service.cc:1514] Processing DeleteTablet for tablet c172b277ffd44289b788c957ade3b3cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:33:27 UTC) from {username='kudu', principal='kudu/127.16.157.126@KRBTEST.COM'} at 127.0.0.1:53156
I20260928 06:33:28.001673 19332 tablet_replica.cc:330] T c172b277ffd44289b788c957ade3b3cd P 5cbf45ff2943424594f6a7568d3a93de: stopping tablet replica
I20260928 06:33:28.001756 19332 raft_consensus.cc:2229] T c172b277ffd44289b788c957ade3b3cd P 5cbf45ff2943424594f6a7568d3a93de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:33:28.001806 19332 raft_consensus.cc:2258] T c172b277ffd44289b788c957ade3b3cd P 5cbf45ff2943424594f6a7568d3a93de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:33:28.002080 19332 ts_tablet_manager.cc:1905] T c172b277ffd44289b788c957ade3b3cd P 5cbf45ff2943424594f6a7568d3a93de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:33:28.002101 18878 tablet_service.cc:1514] Processing DeleteTablet for tablet c172b277ffd44289b788c957ade3b3cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:33:27 UTC) from {username='kudu', principal='kudu/127.16.157.126@KRBTEST.COM'} at 127.0.0.1:55714
I20260928 06:33:28.002213 19331 tablet_replica.cc:330] T c172b277ffd44289b788c957ade3b3cd P 25aa964a114141aeb4b2374eaf7cb85f: stopping tablet replica
I20260928 06:33:28.002271 19331 raft_consensus.cc:2229] T c172b277ffd44289b788c957ade3b3cd P 25aa964a114141aeb4b2374eaf7cb85f [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:33:28.002311 19331 raft_consensus.cc:2258] T c172b277ffd44289b788c957ade3b3cd P 25aa964a114141aeb4b2374eaf7cb85f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:33:28.002501 19330 ts_tablet_manager.cc:1918] T c172b277ffd44289b788c957ade3b3cd P bffb4fdc681a4dd39a816b4a9706a7af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:33:28.002552 19330 log.cc:1196] T c172b277ffd44289b788c957ade3b3cd P bffb4fdc681a4dd39a816b4a9706a7af: Deleting WAL directory at /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/ts-1/wal/wals/c172b277ffd44289b788c957ade3b3cd
I20260928 06:33:28.002568 19331 ts_tablet_manager.cc:1905] T c172b277ffd44289b788c957ade3b3cd P 25aa964a114141aeb4b2374eaf7cb85f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:33:28.002766 19330 ts_tablet_manager.cc:1939] T c172b277ffd44289b788c957ade3b3cd P bffb4fdc681a4dd39a816b4a9706a7af: Deleting consensus metadata
I20260928 06:33:28.003489 18727 catalog_manager.cc:4841] TS bffb4fdc681a4dd39a816b4a9706a7af (127.16.157.66:41079): tablet c172b277ffd44289b788c957ade3b3cd (table test-table [id=2b97f96a36c04dc39e14c904be5a6bb1]) successfully deleted
I20260928 06:33:28.003847 19332 ts_tablet_manager.cc:1918] T c172b277ffd44289b788c957ade3b3cd P 5cbf45ff2943424594f6a7568d3a93de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:33:28.003899 19332 log.cc:1196] T c172b277ffd44289b788c957ade3b3cd P 5cbf45ff2943424594f6a7568d3a93de: Deleting WAL directory at /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/ts-2/wal/wals/c172b277ffd44289b788c957ade3b3cd
I20260928 06:33:28.004078 19331 ts_tablet_manager.cc:1918] T c172b277ffd44289b788c957ade3b3cd P 25aa964a114141aeb4b2374eaf7cb85f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:33:28.004130 19331 log.cc:1196] T c172b277ffd44289b788c957ade3b3cd P 25aa964a114141aeb4b2374eaf7cb85f: Deleting WAL directory at /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/ts-0/wal/wals/c172b277ffd44289b788c957ade3b3cd
I20260928 06:33:28.004118 19332 ts_tablet_manager.cc:1939] T c172b277ffd44289b788c957ade3b3cd P 5cbf45ff2943424594f6a7568d3a93de: Deleting consensus metadata
I20260928 06:33:28.004153 19019 tablet_service.cc:1467] Processing CreateTablet for tablet ff59725d7bba4c6a80d8293befe22f28 (DEFAULT_TABLE table=test-table [id=e3d0279854314fb8b09fd5cb2d96fd04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:33:28.004249 19019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff59725d7bba4c6a80d8293befe22f28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:28.004278 19147 tablet_service.cc:1467] Processing CreateTablet for tablet ff59725d7bba4c6a80d8293befe22f28 (DEFAULT_TABLE table=test-table [id=e3d0279854314fb8b09fd5cb2d96fd04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:33:28.004348 19331 ts_tablet_manager.cc:1939] T c172b277ffd44289b788c957ade3b3cd P 25aa964a114141aeb4b2374eaf7cb85f: Deleting consensus metadata
I20260928 06:33:28.004370 19147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff59725d7bba4c6a80d8293befe22f28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:28.005527 19341 tablet_bootstrap.cc:492] T ff59725d7bba4c6a80d8293befe22f28 P 5cbf45ff2943424594f6a7568d3a93de: Bootstrap starting.
I20260928 06:33:28.005626 19342 tablet_bootstrap.cc:492] T ff59725d7bba4c6a80d8293befe22f28 P bffb4fdc681a4dd39a816b4a9706a7af: Bootstrap starting.
I20260928 06:33:28.005692 18728 catalog_manager.cc:4841] TS 5cbf45ff2943424594f6a7568d3a93de (127.16.157.67:35149): tablet c172b277ffd44289b788c957ade3b3cd (table test-table [id=2b97f96a36c04dc39e14c904be5a6bb1]) successfully deleted
I20260928 06:33:28.005884 19341 tablet_bootstrap.cc:654] T ff59725d7bba4c6a80d8293befe22f28 P 5cbf45ff2943424594f6a7568d3a93de: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:28.005918 18878 tablet_service.cc:1467] Processing CreateTablet for tablet ff59725d7bba4c6a80d8293befe22f28 (DEFAULT_TABLE table=test-table [id=e3d0279854314fb8b09fd5cb2d96fd04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:33:28.006026 18878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff59725d7bba4c6a80d8293befe22f28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:28.006196 19342 tablet_bootstrap.cc:654] T ff59725d7bba4c6a80d8293befe22f28 P bffb4fdc681a4dd39a816b4a9706a7af: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:28.006412 19341 tablet_bootstrap.cc:492] T ff59725d7bba4c6a80d8293befe22f28 P 5cbf45ff2943424594f6a7568d3a93de: No bootstrap required, opened a new log
I20260928 06:33:28.006466 19341 ts_tablet_manager.cc:1397] T ff59725d7bba4c6a80d8293befe22f28 P 5cbf45ff2943424594f6a7568d3a93de: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:28.006568 19341 raft_consensus.cc:348] T ff59725d7bba4c6a80d8293befe22f28 P 5cbf45ff2943424594f6a7568d3a93de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25aa964a114141aeb4b2374eaf7cb85f" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 46219 } } peers { permanent_uuid: "bffb4fdc681a4dd39a816b4a9706a7af" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 41079 } } peers { permanent_uuid: "5cbf45ff2943424594f6a7568d3a93de" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35149 } }
I20260928 06:33:28.006659 19341 raft_consensus.cc:374] T ff59725d7bba4c6a80d8293befe22f28 P 5cbf45ff2943424594f6a7568d3a93de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:28.006693 19341 raft_consensus.cc:729] T ff59725d7bba4c6a80d8293befe22f28 P 5cbf45ff2943424594f6a7568d3a93de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cbf45ff2943424594f6a7568d3a93de, State: Initialized, Role: FOLLOWER
I20260928 06:33:28.006752 19341 consensus_queue.cc:260] T ff59725d7bba4c6a80d8293befe22f28 P 5cbf45ff2943424594f6a7568d3a93de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25aa964a114141aeb4b2374eaf7cb85f" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 46219 } } peers { permanent_uuid: "bffb4fdc681a4dd39a816b4a9706a7af" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 41079 } } peers { permanent_uuid: "5cbf45ff2943424594f6a7568d3a93de" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35149 } }
I20260928 06:33:28.006831 19342 tablet_bootstrap.cc:492] T ff59725d7bba4c6a80d8293befe22f28 P bffb4fdc681a4dd39a816b4a9706a7af: No bootstrap required, opened a new log
I20260928 06:33:28.006886 19342 ts_tablet_manager.cc:1397] T ff59725d7bba4c6a80d8293befe22f28 P bffb4fdc681a4dd39a816b4a9706a7af: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:28.006953 19341 ts_tablet_manager.cc:1428] T ff59725d7bba4c6a80d8293befe22f28 P 5cbf45ff2943424594f6a7568d3a93de: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 06:33:28.007016 19342 raft_consensus.cc:348] T ff59725d7bba4c6a80d8293befe22f28 P bffb4fdc681a4dd39a816b4a9706a7af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25aa964a114141aeb4b2374eaf7cb85f" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 46219 } } peers { permanent_uuid: "bffb4fdc681a4dd39a816b4a9706a7af" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 41079 } } peers { permanent_uuid: "5cbf45ff2943424594f6a7568d3a93de" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35149 } }
I20260928 06:33:28.007084 19342 raft_consensus.cc:374] T ff59725d7bba4c6a80d8293befe22f28 P bffb4fdc681a4dd39a816b4a9706a7af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:28.007104 19342 raft_consensus.cc:729] T ff59725d7bba4c6a80d8293befe22f28 P bffb4fdc681a4dd39a816b4a9706a7af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bffb4fdc681a4dd39a816b4a9706a7af, State: Initialized, Role: FOLLOWER
I20260928 06:33:28.007158 19342 consensus_queue.cc:260] T ff59725d7bba4c6a80d8293befe22f28 P bffb4fdc681a4dd39a816b4a9706a7af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25aa964a114141aeb4b2374eaf7cb85f" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 46219 } } peers { permanent_uuid: "bffb4fdc681a4dd39a816b4a9706a7af" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 41079 } } peers { permanent_uuid: "5cbf45ff2943424594f6a7568d3a93de" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35149 } }
I20260928 06:33:28.007252 19342 ts_tablet_manager.cc:1428] T ff59725d7bba4c6a80d8293befe22f28 P bffb4fdc681a4dd39a816b4a9706a7af: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 06:33:28.007406 18728 catalog_manager.cc:4841] TS 25aa964a114141aeb4b2374eaf7cb85f (127.16.157.65:46219): tablet c172b277ffd44289b788c957ade3b3cd (table test-table [id=2b97f96a36c04dc39e14c904be5a6bb1]) successfully deleted
I20260928 06:33:28.007447 19343 tablet_bootstrap.cc:492] T ff59725d7bba4c6a80d8293befe22f28 P 25aa964a114141aeb4b2374eaf7cb85f: Bootstrap starting.
I20260928 06:33:28.007772 19343 tablet_bootstrap.cc:654] T ff59725d7bba4c6a80d8293befe22f28 P 25aa964a114141aeb4b2374eaf7cb85f: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:28.008378 19343 tablet_bootstrap.cc:492] T ff59725d7bba4c6a80d8293befe22f28 P 25aa964a114141aeb4b2374eaf7cb85f: No bootstrap required, opened a new log
I20260928 06:33:28.008420 19343 ts_tablet_manager.cc:1397] T ff59725d7bba4c6a80d8293befe22f28 P 25aa964a114141aeb4b2374eaf7cb85f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:28.008492 19343 raft_consensus.cc:348] T ff59725d7bba4c6a80d8293befe22f28 P 25aa964a114141aeb4b2374eaf7cb85f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25aa964a114141aeb4b2374eaf7cb85f" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 46219 } } peers { permanent_uuid: "bffb4fdc681a4dd39a816b4a9706a7af" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 41079 } } peers { permanent_uuid: "5cbf45ff2943424594f6a7568d3a93de" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35149 } }
I20260928 06:33:28.008561 19343 raft_consensus.cc:374] T ff59725d7bba4c6a80d8293befe22f28 P 25aa964a114141aeb4b2374eaf7cb85f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:28.008590 19343 raft_consensus.cc:729] T ff59725d7bba4c6a80d8293befe22f28 P 25aa964a114141aeb4b2374eaf7cb85f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25aa964a114141aeb4b2374eaf7cb85f, State: Initialized, Role: FOLLOWER
I20260928 06:33:28.008632 19343 consensus_queue.cc:260] T ff59725d7bba4c6a80d8293befe22f28 P 25aa964a114141aeb4b2374eaf7cb85f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25aa964a114141aeb4b2374eaf7cb85f" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 46219 } } peers { permanent_uuid: "bffb4fdc681a4dd39a816b4a9706a7af" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 41079 } } peers { permanent_uuid: "5cbf45ff2943424594f6a7568d3a93de" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35149 } }
I20260928 06:33:28.008731 19343 ts_tablet_manager.cc:1428] T ff59725d7bba4c6a80d8293befe22f28 P 25aa964a114141aeb4b2374eaf7cb85f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:28.122666 19320 raft_consensus.cc:482] T ff59725d7bba4c6a80d8293befe22f28 P 5cbf45ff2943424594f6a7568d3a93de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:33:28.122769 19320 raft_consensus.cc:504] T ff59725d7bba4c6a80d8293befe22f28 P 5cbf45ff2943424594f6a7568d3a93de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25aa964a114141aeb4b2374eaf7cb85f" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 46219 } } peers { permanent_uuid: "bffb4fdc681a4dd39a816b4a9706a7af" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 41079 } } peers { permanent_uuid: "5cbf45ff2943424594f6a7568d3a93de" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35149 } }
I20260928 06:33:28.122954 19320 leader_election.cc:288] T ff59725d7bba4c6a80d8293befe22f28 P 5cbf45ff2943424594f6a7568d3a93de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 25aa964a114141aeb4b2374eaf7cb85f (127.16.157.65:46219), bffb4fdc681a4dd39a816b4a9706a7af (127.16.157.66:41079)
I20260928 06:33:28.123103 19038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff59725d7bba4c6a80d8293befe22f28" candidate_uuid: "5cbf45ff2943424594f6a7568d3a93de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bffb4fdc681a4dd39a816b4a9706a7af" is_pre_election: true
I20260928 06:33:28.123154 18899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff59725d7bba4c6a80d8293befe22f28" candidate_uuid: "5cbf45ff2943424594f6a7568d3a93de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25aa964a114141aeb4b2374eaf7cb85f" is_pre_election: true
I20260928 06:33:28.123200 19038 raft_consensus.cc:2454] T ff59725d7bba4c6a80d8293befe22f28 P bffb4fdc681a4dd39a816b4a9706a7af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5cbf45ff2943424594f6a7568d3a93de in term 0.
I20260928 06:33:28.123234 18899 raft_consensus.cc:2454] T ff59725d7bba4c6a80d8293befe22f28 P 25aa964a114141aeb4b2374eaf7cb85f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5cbf45ff2943424594f6a7568d3a93de in term 0.
I20260928 06:33:28.123383 19111 leader_election.cc:302] T ff59725d7bba4c6a80d8293befe22f28 P 5cbf45ff2943424594f6a7568d3a93de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cbf45ff2943424594f6a7568d3a93de, bffb4fdc681a4dd39a816b4a9706a7af; no voters:
I20260928 06:33:28.123533 19320 raft_consensus.cc:2781] T ff59725d7bba4c6a80d8293befe22f28 P 5cbf45ff2943424594f6a7568d3a93de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:33:28.123605 19320 raft_consensus.cc:482] T ff59725d7bba4c6a80d8293befe22f28 P 5cbf45ff2943424594f6a7568d3a93de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:33:28.123643 19320 raft_consensus.cc:3037] T ff59725d7bba4c6a80d8293befe22f28 P 5cbf45ff2943424594f6a7568d3a93de [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:28.124281 19320 raft_consensus.cc:504] T ff59725d7bba4c6a80d8293befe22f28 P 5cbf45ff2943424594f6a7568d3a93de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25aa964a114141aeb4b2374eaf7cb85f" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 46219 } } peers { permanent_uuid: "bffb4fdc681a4dd39a816b4a9706a7af" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 41079 } } peers { permanent_uuid: "5cbf45ff2943424594f6a7568d3a93de" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35149 } }
I20260928 06:33:28.124416 19320 leader_election.cc:288] T ff59725d7bba4c6a80d8293befe22f28 P 5cbf45ff2943424594f6a7568d3a93de [CANDIDATE]: Term 1 election: Requested vote from peers 25aa964a114141aeb4b2374eaf7cb85f (127.16.157.65:46219), bffb4fdc681a4dd39a816b4a9706a7af (127.16.157.66:41079)
I20260928 06:33:28.124549 19038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff59725d7bba4c6a80d8293befe22f28" candidate_uuid: "5cbf45ff2943424594f6a7568d3a93de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bffb4fdc681a4dd39a816b4a9706a7af"
I20260928 06:33:28.124580 18899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff59725d7bba4c6a80d8293befe22f28" candidate_uuid: "5cbf45ff2943424594f6a7568d3a93de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25aa964a114141aeb4b2374eaf7cb85f"
I20260928 06:33:28.124634 19038 raft_consensus.cc:3037] T ff59725d7bba4c6a80d8293befe22f28 P bffb4fdc681a4dd39a816b4a9706a7af [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:28.124650 18899 raft_consensus.cc:3037] T ff59725d7bba4c6a80d8293befe22f28 P 25aa964a114141aeb4b2374eaf7cb85f [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:28.125267 18899 raft_consensus.cc:2454] T ff59725d7bba4c6a80d8293befe22f28 P 25aa964a114141aeb4b2374eaf7cb85f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cbf45ff2943424594f6a7568d3a93de in term 1.
I20260928 06:33:28.125392 19038 raft_consensus.cc:2454] T ff59725d7bba4c6a80d8293befe22f28 P bffb4fdc681a4dd39a816b4a9706a7af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cbf45ff2943424594f6a7568d3a93de in term 1.
I20260928 06:33:28.125434 19112 leader_election.cc:302] T ff59725d7bba4c6a80d8293befe22f28 P 5cbf45ff2943424594f6a7568d3a93de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25aa964a114141aeb4b2374eaf7cb85f, 5cbf45ff2943424594f6a7568d3a93de; no voters:
I20260928 06:33:28.125526 19320 raft_consensus.cc:2781] T ff59725d7bba4c6a80d8293befe22f28 P 5cbf45ff2943424594f6a7568d3a93de [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:33:28.125587 19320 raft_consensus.cc:686] T ff59725d7bba4c6a80d8293befe22f28 P 5cbf45ff2943424594f6a7568d3a93de [term 1 LEADER]: Becoming Leader. State: Replica: 5cbf45ff2943424594f6a7568d3a93de, State: Running, Role: LEADER
I20260928 06:33:28.125651 19320 consensus_queue.cc:237] T ff59725d7bba4c6a80d8293befe22f28 P 5cbf45ff2943424594f6a7568d3a93de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25aa964a114141aeb4b2374eaf7cb85f" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 46219 } } peers { permanent_uuid: "bffb4fdc681a4dd39a816b4a9706a7af" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 41079 } } peers { permanent_uuid: "5cbf45ff2943424594f6a7568d3a93de" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35149 } }
I20260928 06:33:28.126303 18741 catalog_manager.cc:5495] T ff59725d7bba4c6a80d8293befe22f28 P 5cbf45ff2943424594f6a7568d3a93de reported cstate change: term changed from 0 to 1, leader changed from <none> to 5cbf45ff2943424594f6a7568d3a93de (127.16.157.67). New cstate: current_term: 1 leader_uuid: "5cbf45ff2943424594f6a7568d3a93de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25aa964a114141aeb4b2374eaf7cb85f" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 46219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bffb4fdc681a4dd39a816b4a9706a7af" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 41079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cbf45ff2943424594f6a7568d3a93de" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35149 } health_report { overall_health: HEALTHY } } }
I20260928 06:33:28.155498 19359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:28.152927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53168 (local address 127.16.157.67:35149)
0928 06:33:28.153099 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:28.153101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:28.153108 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:33:28.153127 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:28.153127 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:28.153137 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:28.153195 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:33:28.153298 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:28.153931 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:28.154723 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:28.155005 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:28.155061 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:28.155219 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:33:28.155248 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:33:28.155314 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:28.155368 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 06:33:28.155390 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":105,"threads_started":1}
I20260928 06:33:28.157828 19038 raft_consensus.cc:1261] T ff59725d7bba4c6a80d8293befe22f28 P bffb4fdc681a4dd39a816b4a9706a7af [term 1 FOLLOWER]: Refusing update from remote peer 5cbf45ff2943424594f6a7568d3a93de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:33:28.157841 18899 raft_consensus.cc:1261] T ff59725d7bba4c6a80d8293befe22f28 P 25aa964a114141aeb4b2374eaf7cb85f [term 1 FOLLOWER]: Refusing update from remote peer 5cbf45ff2943424594f6a7568d3a93de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:33:28.158134 19324 consensus_queue.cc:1035] T ff59725d7bba4c6a80d8293befe22f28 P 5cbf45ff2943424594f6a7568d3a93de [LEADER]: Connected to new peer: Peer: permanent_uuid: "bffb4fdc681a4dd39a816b4a9706a7af" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 41079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:33:28.158283 19324 consensus_queue.cc:1035] T ff59725d7bba4c6a80d8293befe22f28 P 5cbf45ff2943424594f6a7568d3a93de [LEADER]: Connected to new peer: Peer: permanent_uuid: "25aa964a114141aeb4b2374eaf7cb85f" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 46219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 06:33:28.162688 19313 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 06:33:29.681293 18740 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49228:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:33:29.681406 18740 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49228:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:33:29.682452 18740 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d9646455aab14be695ee9033c14d8ac0: Sending DeleteTablet for 3 replicas of tablet ff59725d7bba4c6a80d8293befe22f28
I20260928 06:33:29.682724 19019 tablet_service.cc:1514] Processing DeleteTablet for tablet ff59725d7bba4c6a80d8293befe22f28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:33:29 UTC) from {username='kudu', principal='kudu/127.16.157.126@KRBTEST.COM'} at 127.0.0.1:39538
I20260928 06:33:29.682777 18878 tablet_service.cc:1514] Processing DeleteTablet for tablet ff59725d7bba4c6a80d8293befe22f28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:33:29 UTC) from {username='kudu', principal='kudu/127.16.157.126@KRBTEST.COM'} at 127.0.0.1:55714
I20260928 06:33:29.682895 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 18814
I20260928 06:33:29.682928 19386 tablet_replica.cc:330] T ff59725d7bba4c6a80d8293befe22f28 P bffb4fdc681a4dd39a816b4a9706a7af: stopping tablet replica
I20260928 06:33:29.682979 19386 raft_consensus.cc:2229] T ff59725d7bba4c6a80d8293befe22f28 P bffb4fdc681a4dd39a816b4a9706a7af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:33:29.684136 19386 raft_consensus.cc:2258] T ff59725d7bba4c6a80d8293befe22f28 P bffb4fdc681a4dd39a816b4a9706a7af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:33:29.684321 19147 tablet_service.cc:1514] Processing DeleteTablet for tablet ff59725d7bba4c6a80d8293befe22f28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:33:29 UTC) from {username='kudu', principal='kudu/127.16.157.126@KRBTEST.COM'} at 127.0.0.1:53156
I20260928 06:33:29.684392 19386 ts_tablet_manager.cc:1905] T ff59725d7bba4c6a80d8293befe22f28 P bffb4fdc681a4dd39a816b4a9706a7af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:33:29.684517 19388 tablet_replica.cc:330] T ff59725d7bba4c6a80d8293befe22f28 P 5cbf45ff2943424594f6a7568d3a93de: stopping tablet replica
I20260928 06:33:29.684572 19388 raft_consensus.cc:2229] T ff59725d7bba4c6a80d8293befe22f28 P 5cbf45ff2943424594f6a7568d3a93de [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:33:29.684625 19388 raft_consensus.cc:2258] T ff59725d7bba4c6a80d8293befe22f28 P 5cbf45ff2943424594f6a7568d3a93de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:33:29.684842 19388 ts_tablet_manager.cc:1905] T ff59725d7bba4c6a80d8293befe22f28 P 5cbf45ff2943424594f6a7568d3a93de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:33:29.686535 19386 ts_tablet_manager.cc:1918] T ff59725d7bba4c6a80d8293befe22f28 P bffb4fdc681a4dd39a816b4a9706a7af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 06:33:29.686661 19386 log.cc:1196] T ff59725d7bba4c6a80d8293befe22f28 P bffb4fdc681a4dd39a816b4a9706a7af: Deleting WAL directory at /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/ts-1/wal/wals/ff59725d7bba4c6a80d8293befe22f28
I20260928 06:33:29.686985 19386 ts_tablet_manager.cc:1939] T ff59725d7bba4c6a80d8293befe22f28 P bffb4fdc681a4dd39a816b4a9706a7af: Deleting consensus metadata
I20260928 06:33:29.687019 19388 ts_tablet_manager.cc:1918] T ff59725d7bba4c6a80d8293befe22f28 P 5cbf45ff2943424594f6a7568d3a93de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 06:33:29.687065 19388 log.cc:1196] T ff59725d7bba4c6a80d8293befe22f28 P 5cbf45ff2943424594f6a7568d3a93de: Deleting WAL directory at /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577196946045-17013-0/minicluster-data/ts-2/wal/wals/ff59725d7bba4c6a80d8293befe22f28
I20260928 06:33:29.687315 19388 ts_tablet_manager.cc:1939] T ff59725d7bba4c6a80d8293befe22f28 P 5cbf45ff2943424594f6a7568d3a93de: Deleting consensus metadata
I20260928 06:33:29.687471 18727 catalog_manager.cc:4841] TS bffb4fdc681a4dd39a816b4a9706a7af (127.16.157.66:41079): tablet ff59725d7bba4c6a80d8293befe22f28 (table test-table [id=e3d0279854314fb8b09fd5cb2d96fd04]) successfully deleted
I20260928 06:33:29.689782 18728 catalog_manager.cc:4841] TS 5cbf45ff2943424594f6a7568d3a93de (127.16.157.67:35149): tablet ff59725d7bba4c6a80d8293befe22f28 (table test-table [id=e3d0279854314fb8b09fd5cb2d96fd04]) successfully deleted
W20260928 06:33:29.692596 18728 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.157.65:46219 (error 108)
I20260928 06:33:29.692901 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 18954
I20260928 06:33:29.696448 19390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:29.695974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.65:46219 (local address 127.0.0.1:39242)
0928 06:33:29.696132 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:29.696188 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.65:46219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":75,"threads_started":1}
W20260928 06:33:29.698634 18728 catalog_manager.cc:4584] TS 25aa964a114141aeb4b2374eaf7cb85f (127.16.157.65:46219): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ff59725d7bba4c6a80d8293befe22f28: Network error: Client connection negotiation failed: client connection to 127.16.157.65:46219: connect: Connection refused (error 111)
I20260928 06:33:29.703003 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 19094
I20260928 06:33:29.712018 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 18710
2026-09-28T06:33:29Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5837 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:33:29 dist-test-slave-rrj5 krb5kdc[19393](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:33:29 dist-test-slave-rrj5 krb5kdc[19393](info): set up 2 sockets
Sep 28 06:33:29 dist-test-slave-rrj5 krb5kdc[19393](info): commencing operation
krb5kdc: starting...
W20260928 06:33:30.946656 17013 mini_kdc.cc:121] Time spent starting KDC: real 1.208s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Sep 28 06:33:31 dist-test-slave-rrj5 krb5kdc[19393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:33:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:33:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.157.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab.
Entry for principal kudu/127.16.157.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab.
I20260928 06:33:31.080189 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:45227
--webserver_interface=127.16.157.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=kudu/127.16.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:41681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.157.126:45227
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:31.167238 19406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:31.167465 19406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:31.167502 19406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:31.168655 19406 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:33:31.168720 19406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:31.168746 19406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:31.168779 19406 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:33:31.168809 19406 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:33:31.170406 19406 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:41681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.157.126:45227
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=kudu/127.16.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:45227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:31.170670 19406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:31.170938 19406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:31.176575 19411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:31.176550 19414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:31.176565 19412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:31.177234 19406 server_base.cc:1034] running on GCE node
I20260928 06:33:31.177484 19406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:31.177803 19406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:31.178977 19406 hybrid_clock.cc:648] HybridClock initialized: now 1790577211178965 us; error 32 us; skew 500 ppm
Sep 28 06:33:31 dist-test-slave-rrj5 krb5kdc[19393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577211, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:31.181409 19406 init.cc:377] Logged in from keytab as kudu/127.16.157.126@KRBTEST.COM (short username kudu)
I20260928 06:33:31.181938 19406 webserver.cc:460] Webserver started at http://127.16.157.126:46785/ using document root <none> and password file <none>
I20260928 06:33:31.182168 19406 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:31.182233 19406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:31.182386 19406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:31.183429 19406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/minicluster-data/master-0/data/instance:
uuid: "c197f53df70e4bb3a5a7cd0c8039b55c"
format_stamp: "Formatted at 2026-09-28 06:33:31 on dist-test-slave-rrj5"
server_key: "ea08a8c7ae1be1c41dfdfe7b1f4c7023"
server_key_iv: "e5b0530e891dc415ed4f2be50aeec5fc"
server_key_version: "encryptionkey@0"
I20260928 06:33:31.183892 19406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/minicluster-data/master-0/wal/instance:
uuid: "c197f53df70e4bb3a5a7cd0c8039b55c"
format_stamp: "Formatted at 2026-09-28 06:33:31 on dist-test-slave-rrj5"
server_key: "ea08a8c7ae1be1c41dfdfe7b1f4c7023"
server_key_iv: "e5b0530e891dc415ed4f2be50aeec5fc"
server_key_version: "encryptionkey@0"
I20260928 06:33:31.185472 19406 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:33:31.186300 19421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:31.186471 19406 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:33:31.186558 19406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/minicluster-data/master-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/minicluster-data/master-0/wal
uuid: "c197f53df70e4bb3a5a7cd0c8039b55c"
format_stamp: "Formatted at 2026-09-28 06:33:31 on dist-test-slave-rrj5"
server_key: "ea08a8c7ae1be1c41dfdfe7b1f4c7023"
server_key_iv: "e5b0530e891dc415ed4f2be50aeec5fc"
server_key_version: "encryptionkey@0"
I20260928 06:33:31.186655 19406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:31.200532 19406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:31.203058 19406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:31.203224 19406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:31.207892 19406 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.126:45227
I20260928 06:33:31.207957 19473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.126:45227 every 8 connection(s)
I20260928 06:33:31.208427 19406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
I20260928 06:33:31.209436 19474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:31.212777 19474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c197f53df70e4bb3a5a7cd0c8039b55c: Bootstrap starting.
I20260928 06:33:31.213521 19474 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c197f53df70e4bb3a5a7cd0c8039b55c: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:31.213907 19474 log.cc:824] T 00000000000000000000000000000000 P c197f53df70e4bb3a5a7cd0c8039b55c: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:31.214809 19474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c197f53df70e4bb3a5a7cd0c8039b55c: No bootstrap required, opened a new log
I20260928 06:33:31.215298 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 19406
I20260928 06:33:31.215382 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/minicluster-data/master-0/wal/instance
I20260928 06:33:31.215533 17013 external_mini_cluster.cc:1428] Setting key c02282ed8431cbee37d7d45135665a09
I20260928 06:33:31.217339 19474 raft_consensus.cc:348] T 00000000000000000000000000000000 P c197f53df70e4bb3a5a7cd0c8039b55c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c197f53df70e4bb3a5a7cd0c8039b55c" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 45227 } }
I20260928 06:33:31.217581 19474 raft_consensus.cc:374] T 00000000000000000000000000000000 P c197f53df70e4bb3a5a7cd0c8039b55c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:31.217738 19474 raft_consensus.cc:729] T 00000000000000000000000000000000 P c197f53df70e4bb3a5a7cd0c8039b55c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c197f53df70e4bb3a5a7cd0c8039b55c, State: Initialized, Role: FOLLOWER
I20260928 06:33:31.217970 19474 consensus_queue.cc:260] T 00000000000000000000000000000000 P c197f53df70e4bb3a5a7cd0c8039b55c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c197f53df70e4bb3a5a7cd0c8039b55c" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 45227 } }
I20260928 06:33:31.218118 19474 raft_consensus.cc:388] T 00000000000000000000000000000000 P c197f53df70e4bb3a5a7cd0c8039b55c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:33:31.218199 19474 raft_consensus.cc:482] T 00000000000000000000000000000000 P c197f53df70e4bb3a5a7cd0c8039b55c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:33:31.218302 19474 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c197f53df70e4bb3a5a7cd0c8039b55c [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:31.219046 19474 raft_consensus.cc:504] T 00000000000000000000000000000000 P c197f53df70e4bb3a5a7cd0c8039b55c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c197f53df70e4bb3a5a7cd0c8039b55c" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 45227 } }
I20260928 06:33:31.219209 19474 leader_election.cc:302] T 00000000000000000000000000000000 P c197f53df70e4bb3a5a7cd0c8039b55c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c197f53df70e4bb3a5a7cd0c8039b55c; no voters:
I20260928 06:33:31.219380 19474 leader_election.cc:288] T 00000000000000000000000000000000 P c197f53df70e4bb3a5a7cd0c8039b55c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:33:31.219584 19480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c197f53df70e4bb3a5a7cd0c8039b55c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:33:31.219707 19480 raft_consensus.cc:686] T 00000000000000000000000000000000 P c197f53df70e4bb3a5a7cd0c8039b55c [term 1 LEADER]: Becoming Leader. State: Replica: c197f53df70e4bb3a5a7cd0c8039b55c, State: Running, Role: LEADER
I20260928 06:33:31.219792 19480 consensus_queue.cc:237] T 00000000000000000000000000000000 P c197f53df70e4bb3a5a7cd0c8039b55c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c197f53df70e4bb3a5a7cd0c8039b55c" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 45227 } }
I20260928 06:33:31.219825 19474 sys_catalog.cc:564] T 00000000000000000000000000000000 P c197f53df70e4bb3a5a7cd0c8039b55c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:33:31.220108 19480 sys_catalog.cc:455] T 00000000000000000000000000000000 P c197f53df70e4bb3a5a7cd0c8039b55c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c197f53df70e4bb3a5a7cd0c8039b55c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c197f53df70e4bb3a5a7cd0c8039b55c" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 45227 } } }
I20260928 06:33:31.220198 19480 sys_catalog.cc:458] T 00000000000000000000000000000000 P c197f53df70e4bb3a5a7cd0c8039b55c [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:31.220323 19480 sys_catalog.cc:455] T 00000000000000000000000000000000 P c197f53df70e4bb3a5a7cd0c8039b55c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c197f53df70e4bb3a5a7cd0c8039b55c. Latest consensus state: current_term: 1 leader_uuid: "c197f53df70e4bb3a5a7cd0c8039b55c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c197f53df70e4bb3a5a7cd0c8039b55c" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 45227 } } }
I20260928 06:33:31.220384 19480 sys_catalog.cc:458] T 00000000000000000000000000000000 P c197f53df70e4bb3a5a7cd0c8039b55c [sys.catalog]: This master's current role is: LEADER
W20260928 06:33:31.222139 19494 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c197f53df70e4bb3a5a7cd0c8039b55c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:33:31.222208 19494 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:33:31.222662 19483 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:33:31.222918 19483 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:33:31.224941 19483 catalog_manager.cc:1347] Generated new cluster ID: ea601e84f72146b7ae34eec5131202e2
I20260928 06:33:31.225037 19483 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:33:31 dist-test-slave-rrj5 krb5kdc[19393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:33:31.231156 19478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:31.216159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34238 (local address 127.16.157.126:45227)
0928 06:33:31.216355 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:31.216365 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:31.216396 (+ 31us) server_negotiation.cc:407] Connection header received
0928 06:33:31.216902 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:31.216909 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:31.216969 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:31.217887 (+ 918us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:31.218018 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:31.219741 (+ 1723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:31.220867 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:31.221465 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:31.221924 (+ 459us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:31.227227 (+ 5303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:31.227231 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:31.227231 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:31.227244 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:31.229110 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:31.229532 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:31.229535 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:31.229535 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:31.229714 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:31.229920 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:31.229920 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:31.229921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:31.230209 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:31.230347 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:31.230836 (+ 489us) server_negotiation.cc:299] Negotiation successful
0928 06:33:31.230962 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":80,"threads_started":1}
I20260928 06:33:31.233074 19483 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:33:31.233708 19483 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:33:31.242971 19483 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c197f53df70e4bb3a5a7cd0c8039b55c: Generated new TSK 0
I20260928 06:33:31.243155 19483 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.157.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab.
Entry for principal kudu/127.16.157.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab.
I20260928 06:33:31.280552 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.65:0
--local_ip_for_outbound_sockets=127.16.157.65
--webserver_interface=127.16.157.65
--webserver_port=0
--tserver_master_addrs=127.16.157.126:45227
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab
--principal=kudu/127.16.157.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:41681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:31.385303 19500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:31.385551 19500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:31.385610 19500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:31.387472 19500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:31.387529 19500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:31.387569 19500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.65
I20260928 06:33:31.389545 19500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:41681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab
--principal=kudu/127.16.157.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.157.65
--webserver_port=0
--tserver_master_addrs=127.16.157.126:45227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.65
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:31.389842 19500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:31.390177 19500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:31.395694 19506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:31.396042 19508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:31.395833 19505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:31.396709 19500 server_base.cc:1034] running on GCE node
I20260928 06:33:31.396973 19500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:31.397258 19500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:31.398595 19500 hybrid_clock.cc:648] HybridClock initialized: now 1790577211398572 us; error 35 us; skew 500 ppm
Sep 28 06:33:31 dist-test-slave-rrj5 krb5kdc[19393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577211, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:31.401315 19500 init.cc:377] Logged in from keytab as kudu/127.16.157.65@KRBTEST.COM (short username kudu)
I20260928 06:33:31.401870 19500 webserver.cc:460] Webserver started at http://127.16.157.65:46743/ using document root <none> and password file <none>
I20260928 06:33:31.402109 19500 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:31.402184 19500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:31.402315 19500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:31.403366 19500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-0/data/instance:
uuid: "bae07809630f4fb8b4a8d6831383d86d"
format_stamp: "Formatted at 2026-09-28 06:33:31 on dist-test-slave-rrj5"
server_key: "ee8b1b322dde803ec66dbc9739f066ac"
server_key_iv: "f5a54716f12f5ea261a062707fd8402d"
server_key_version: "encryptionkey@0"
I20260928 06:33:31.403740 19500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-0/wal/instance:
uuid: "bae07809630f4fb8b4a8d6831383d86d"
format_stamp: "Formatted at 2026-09-28 06:33:31 on dist-test-slave-rrj5"
server_key: "ee8b1b322dde803ec66dbc9739f066ac"
server_key_iv: "f5a54716f12f5ea261a062707fd8402d"
server_key_version: "encryptionkey@0"
I20260928 06:33:31.405416 19500 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:33:31.406383 19515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:31.406587 19500 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:33:31.406713 19500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-0/wal
uuid: "bae07809630f4fb8b4a8d6831383d86d"
format_stamp: "Formatted at 2026-09-28 06:33:31 on dist-test-slave-rrj5"
server_key: "ee8b1b322dde803ec66dbc9739f066ac"
server_key_iv: "f5a54716f12f5ea261a062707fd8402d"
server_key_version: "encryptionkey@0"
I20260928 06:33:31.406826 19500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:31.436365 19500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:31.438977 19500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:31.439157 19500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:31.439450 19500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:33:31.439772 19500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:33:31.439827 19500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:31.439896 19500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:33:31.439931 19500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:31.446664 19500 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.65:34225
I20260928 06:33:31.446700 19628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.65:34225 every 8 connection(s)
I20260928 06:33:31.447146 19500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
I20260928 06:33:31.447525 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 19500
I20260928 06:33:31.447614 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-0/wal/instance
I20260928 06:33:31.447775 17013 external_mini_cluster.cc:1428] Setting key c4a1311807f4aa14ec4796bd13da4c86
Sep 28 06:33:31 dist-test-slave-rrj5 krb5kdc[19393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577211, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.65@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.157.66@KRBTEST.COM; defaulting to no policy
I20260928 06:33:31.459197 19478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:31.448174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:56663 (local address 127.16.157.126:45227)
0928 06:33:31.448379 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:31.448381 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:31.448390 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:33:31.448948 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:31.448950 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:31.448963 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:31.449026 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:31.450493 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:31.451021 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:31.452082 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:31.452506 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:31.452593 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:31.455548 (+ 2955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:31.455550 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:31.455551 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:31.455559 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:31.457386 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:31.457986 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:31.457988 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:31.457989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:31.458049 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:31.458404 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:31.458405 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:31.458406 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:31.458536 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:31.458615 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:31.458995 (+ 380us) server_negotiation.cc:299] Negotiation successful
0928 06:33:31.459088 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183}
I20260928 06:33:31.459523 19631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:31.447949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:45227 (local address 127.16.157.65:56663)
0928 06:33:31.448124 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:31.448179 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:31.448824 (+ 645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:31.449166 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:31.449172 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:31.450150 (+ 978us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:31.450380 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:31.450386 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:31.451126 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:31.451127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:31.451955 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:31.451957 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:31.452614 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:31.452615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:31.452735 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:31.453400 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:31.453412 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:31.455419 (+ 2007us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:31.457486 (+ 2067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:31.457493 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:31.457495 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:31.457869 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:31.458131 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:31.458132 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:31.458132 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:31.458325 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:31.458617 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:31.458625 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:31.458786 (+ 161us) client_negotiation.cc:769] Sending connection context
0928 06:33:31.458961 (+ 175us) client_negotiation.cc:240] Negotiation successful
0928 06:33:31.459142 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":80,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.66@KRBTEST.COM" created.
I20260928 06:33:31.460173 19629 heartbeater.cc:347] Connected to a master server at 127.16.157.126:45227
I20260928 06:33:31.460284 19629 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:31.460506 19629 heartbeater.cc:511] Master 127.16.157.126:45227 requested a full tablet report, sending...
I20260928 06:33:31.461077 19438 ts_manager.cc:194] Registered new tserver with Master: bae07809630f4fb8b4a8d6831383d86d (127.16.157.65:34225)
I20260928 06:33:31.461843 19438 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.157.65@KRBTEST.COM'} at 127.16.157.65:56663
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab.
Entry for principal kudu/127.16.157.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab.
I20260928 06:33:31.475992 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.66:0
--local_ip_for_outbound_sockets=127.16.157.66
--webserver_interface=127.16.157.66
--webserver_port=0
--tserver_master_addrs=127.16.157.126:45227
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab
--principal=kudu/127.16.157.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:41681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:31.567239 19634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:31.567489 19634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:31.567520 19634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:31.568964 19634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:31.569031 19634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:31.569077 19634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.66
I20260928 06:33:31.571233 19634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:41681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab
--principal=kudu/127.16.157.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.157.66
--webserver_port=0
--tserver_master_addrs=127.16.157.126:45227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.66
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:31.571485 19634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:31.571784 19634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:31.577090 19639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:31.577199 19640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:31.577085 19642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:31.577245 19634 server_base.cc:1034] running on GCE node
I20260928 06:33:31.577535 19634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:31.577841 19634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:31.579006 19634 hybrid_clock.cc:648] HybridClock initialized: now 1790577211578976 us; error 48 us; skew 500 ppm
Sep 28 06:33:31 dist-test-slave-rrj5 krb5kdc[19393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577211, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:31.581700 19634 init.cc:377] Logged in from keytab as kudu/127.16.157.66@KRBTEST.COM (short username kudu)
I20260928 06:33:31.582224 19634 webserver.cc:460] Webserver started at http://127.16.157.66:36799/ using document root <none> and password file <none>
I20260928 06:33:31.582495 19634 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:31.582576 19634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:31.582715 19634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:31.583822 19634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-1/data/instance:
uuid: "55396ae199144eabb005f48c3995fe7e"
format_stamp: "Formatted at 2026-09-28 06:33:31 on dist-test-slave-rrj5"
server_key: "6d33c4ce6c7231cc40c183f0444561ca"
server_key_iv: "5b88178901652e8619db4da00f63f99a"
server_key_version: "encryptionkey@0"
I20260928 06:33:31.584198 19634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-1/wal/instance:
uuid: "55396ae199144eabb005f48c3995fe7e"
format_stamp: "Formatted at 2026-09-28 06:33:31 on dist-test-slave-rrj5"
server_key: "6d33c4ce6c7231cc40c183f0444561ca"
server_key_iv: "5b88178901652e8619db4da00f63f99a"
server_key_version: "encryptionkey@0"
I20260928 06:33:31.585534 19634 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:33:31.586367 19649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:31.586545 19634 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:31.586647 19634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-1/wal
uuid: "55396ae199144eabb005f48c3995fe7e"
format_stamp: "Formatted at 2026-09-28 06:33:31 on dist-test-slave-rrj5"
server_key: "6d33c4ce6c7231cc40c183f0444561ca"
server_key_iv: "5b88178901652e8619db4da00f63f99a"
server_key_version: "encryptionkey@0"
I20260928 06:33:31.586735 19634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:31.611737 19634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:31.613912 19634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:31.614078 19634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:31.614307 19634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:33:31.614619 19634 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:33:31.614675 19634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:31.614754 19634 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:33:31.614784 19634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:31.621102 19634 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.66:32795
I20260928 06:33:31.621181 19762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.66:32795 every 8 connection(s)
I20260928 06:33:31.621490 19634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
Sep 28 06:33:31 dist-test-slave-rrj5 krb5kdc[19393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577211, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.66@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:33:31.631511 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 19634
I20260928 06:33:31.631623 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-1/wal/instance
I20260928 06:33:31.631831 17013 external_mini_cluster.cc:1428] Setting key 4719eee446581be66aeba9da6e6f4be0
I20260928 06:33:31.631886 19478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:31.622249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.66:53079 (local address 127.16.157.126:45227)
0928 06:33:31.622298 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:31.622301 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:31.622532 (+ 231us) server_negotiation.cc:407] Connection header received
0928 06:33:31.622979 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:31.622980 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:31.623002 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:31.623062 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:31.624054 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:31.624578 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:31.626059 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:31.626275 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:31.626361 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:31.628777 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:31.628778 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:31.628779 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:31.628786 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:31.630336 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:31.630762 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:31.630763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:31.630764 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:31.630824 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:31.631098 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:31.631099 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:31.631099 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:31.631232 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:31.631331 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:31.631691 (+ 360us) server_negotiation.cc:299] Negotiation successful
0928 06:33:31.631786 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 06:33:31.632458 19765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:31.622236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:45227 (local address 127.16.157.66:53079)
0928 06:33:31.622412 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:31.622447 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:31.622891 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:31.623113 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:31.623116 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:31.623743 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:31.623929 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:31.623935 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:31.624680 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:31.624681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:31.625947 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:31.625949 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:31.626371 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:31.626372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:31.626495 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:31.627019 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:31.627029 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:31.628648 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:31.630421 (+ 1773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:31.630428 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:31.630430 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:31.630658 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:31.630893 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:31.630894 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:31.630894 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:31.631011 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:31.631334 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:31.631342 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:31.631516 (+ 174us) client_negotiation.cc:769] Sending connection context
0928 06:33:31.632035 (+ 519us) client_negotiation.cc:240] Negotiation successful
0928 06:33:31.632234 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":94,"threads_started":1}
I20260928 06:33:31.633819 19763 heartbeater.cc:347] Connected to a master server at 127.16.157.126:45227
I20260928 06:33:31.633936 19763 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:31.634150 19763 heartbeater.cc:511] Master 127.16.157.126:45227 requested a full tablet report, sending...
I20260928 06:33:31.634582 19438 ts_manager.cc:194] Registered new tserver with Master: 55396ae199144eabb005f48c3995fe7e (127.16.157.66:32795)
I20260928 06:33:31.635155 19438 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.157.66@KRBTEST.COM'} at 127.16.157.66:53079
WARNING: no policy specified for kudu/127.16.157.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab.
Entry for principal kudu/127.16.157.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab.
I20260928 06:33:31.658083 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.67:0
--local_ip_for_outbound_sockets=127.16.157.67
--webserver_interface=127.16.157.67
--webserver_port=0
--tserver_master_addrs=127.16.157.126:45227
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab
--principal=kudu/127.16.157.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:41681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:31.747488 19768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:31.747807 19768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:31.747854 19768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:31.749114 19768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:31.749192 19768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:31.749238 19768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.67
I20260928 06:33:31.750829 19768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:41681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab
--principal=kudu/127.16.157.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.157.67
--webserver_port=0
--tserver_master_addrs=127.16.157.126:45227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.67
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:31.751127 19768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:31.751382 19768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:31.756884 19773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:31.756906 19774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:31.756899 19776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:31.757164 19768 server_base.cc:1034] running on GCE node
I20260928 06:33:31.757380 19768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:31.757690 19768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:31.758868 19768 hybrid_clock.cc:648] HybridClock initialized: now 1790577211758844 us; error 42 us; skew 500 ppm
Sep 28 06:33:31 dist-test-slave-rrj5 krb5kdc[19393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577211, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:31.761844 19768 init.cc:377] Logged in from keytab as kudu/127.16.157.67@KRBTEST.COM (short username kudu)
I20260928 06:33:31.762336 19768 webserver.cc:460] Webserver started at http://127.16.157.67:38955/ using document root <none> and password file <none>
I20260928 06:33:31.762588 19768 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:31.762655 19768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:31.762794 19768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:31.763839 19768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-2/data/instance:
uuid: "464a71cc72194edc8bd5f7dc91c88854"
format_stamp: "Formatted at 2026-09-28 06:33:31 on dist-test-slave-rrj5"
server_key: "c2b888fc4b70afb4aed6776a8c7e1461"
server_key_iv: "d3d486e2d8cfa80a634422e230325308"
server_key_version: "encryptionkey@0"
I20260928 06:33:31.764214 19768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-2/wal/instance:
uuid: "464a71cc72194edc8bd5f7dc91c88854"
format_stamp: "Formatted at 2026-09-28 06:33:31 on dist-test-slave-rrj5"
server_key: "c2b888fc4b70afb4aed6776a8c7e1461"
server_key_iv: "d3d486e2d8cfa80a634422e230325308"
server_key_version: "encryptionkey@0"
I20260928 06:33:31.765738 19768 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 06:33:31.766559 19783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:31.766741 19768 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:31.766830 19768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-2/wal
uuid: "464a71cc72194edc8bd5f7dc91c88854"
format_stamp: "Formatted at 2026-09-28 06:33:31 on dist-test-slave-rrj5"
server_key: "c2b888fc4b70afb4aed6776a8c7e1461"
server_key_iv: "d3d486e2d8cfa80a634422e230325308"
server_key_version: "encryptionkey@0"
I20260928 06:33:31.766911 19768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:31.785243 19768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:31.787681 19768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:31.787847 19768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:31.788079 19768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:33:31.788405 19768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:33:31.788457 19768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:31.788496 19768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:33:31.788522 19768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:31.795583 19768 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.67:37977
I20260928 06:33:31.795619 19896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.67:37977 every 8 connection(s)
I20260928 06:33:31.795933 19768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
Sep 28 06:33:31 dist-test-slave-rrj5 krb5kdc[19393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577211, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.67@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:33:31.803426 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 19768
I20260928 06:33:31.803527 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-2/wal/instance
I20260928 06:33:31.803756 17013 external_mini_cluster.cc:1428] Setting key e892a2d6615a859e84fc5d40a6543e4b
I20260928 06:33:31.806130 19478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:31.796680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:50749 (local address 127.16.157.126:45227)
0928 06:33:31.796722 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:31.796724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:31.796946 (+ 222us) server_negotiation.cc:407] Connection header received
0928 06:33:31.797429 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:31.797430 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:31.797438 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:31.797522 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:31.798675 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:31.799153 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:31.800209 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:31.800457 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:31.800552 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:31.802870 (+ 2318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:31.802872 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:31.802872 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:31.802880 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:31.804719 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:31.805137 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:31.805138 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:31.805139 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:31.805205 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:31.805462 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:31.805463 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:31.805463 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:31.805636 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:31.805698 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:31.805968 (+ 270us) server_negotiation.cc:299] Negotiation successful
0928 06:33:31.806044 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:33:31.806331 19899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:31.796647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:45227 (local address 127.16.157.67:50749)
0928 06:33:31.796822 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:31.796864 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:31.797346 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:31.797632 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:31.797637 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:31.798355 (+ 718us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:31.798543 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:31.798548 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:31.799274 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:31.799275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:31.800107 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:31.800110 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:31.800565 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:31.800566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:31.800712 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:31.801252 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:31.801261 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:31.802761 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:31.804782 (+ 2021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:31.804790 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:31.804793 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:31.805024 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:31.805282 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:31.805283 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:31.805283 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:31.805391 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:31.805691 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:31.805697 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:31.805830 (+ 133us) client_negotiation.cc:769] Sending connection context
0928 06:33:31.805935 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 06:33:31.806094 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":96,"threads_started":1}
I20260928 06:33:31.806916 19897 heartbeater.cc:347] Connected to a master server at 127.16.157.126:45227
I20260928 06:33:31.807025 19897 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:31.807228 19897 heartbeater.cc:511] Master 127.16.157.126:45227 requested a full tablet report, sending...
I20260928 06:33:31.807621 19438 ts_manager.cc:194] Registered new tserver with Master: 464a71cc72194edc8bd5f7dc91c88854 (127.16.157.67:37977)
I20260928 06:33:31.807993 19438 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.157.67@KRBTEST.COM'} at 127.16.157.67:50749
I20260928 06:33:31.815462 17013 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 06:33:31.827139 19478 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:33:31.825996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34252 (local address 127.16.157.126:45227)
0928 06:33:31.826067 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:31.826069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:31.826213 (+ 144us) server_negotiation.cc:407] Connection header received
0928 06:33:31.826290 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:31.826291 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:31.826300 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:31.826350 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:31.827068 (+ 718us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34252: BlockingRecv error: recv got EOF from 127.0.0.1:34252 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 06:33:31.828348 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 19500
I20260928 06:33:31.833740 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 19634
I20260928 06:33:31.838330 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 19768
I20260928 06:33:31.842727 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 19406
2026-09-28T06:33:31Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2126 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577196946045-17013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:33:31 dist-test-slave-rrj5 krb5kdc[19909](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:33:31 dist-test-slave-rrj5 krb5kdc[19909](info): set up 2 sockets
Sep 28 06:33:31 dist-test-slave-rrj5 krb5kdc[19909](info): commencing operation
krb5kdc: starting...
W20260928 06:33:32.931474 17013 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Sep 28 06:33:33 dist-test-slave-rrj5 krb5kdc[19909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:33:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:33:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.157.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab.
Entry for principal kudu/127.16.157.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab.
I20260928 06:33:33.054814 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577196946045-17013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577196946045-17013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577196946045-17013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:44683
--webserver_interface=127.16.157.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=kudu/127.16.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:38765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.157.126:44683
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:33.148557 19922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:33.148773 19922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:33.148800 19922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:33.150043 19922 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:33:33.150089 19922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:33.150106 19922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:33.150121 19922 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:33:33.150156 19922 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:33:33.151710 19922 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:38765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577196946045-17013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577196946045-17013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.157.126:44683
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=kudu/127.16.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:44683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19922
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577196946045-17013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:33.151959 19922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:33.152209 19922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:33.157918 19928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:33.157927 19927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:33.158157 19922 server_base.cc:1034] running on GCE node
W20260928 06:33:33.158288 19930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:33.158540 19922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:33.158856 19922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:33.160023 19922 hybrid_clock.cc:648] HybridClock initialized: now 1790577213159997 us; error 41 us; skew 500 ppm
Sep 28 06:33:33 dist-test-slave-rrj5 krb5kdc[19909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577213, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:33.162860 19922 init.cc:377] Logged in from keytab as kudu/127.16.157.126@KRBTEST.COM (short username kudu)
I20260928 06:33:33.163357 19922 webserver.cc:460] Webserver started at http://127.16.157.126:35391/ using document root <none> and password file <none>
I20260928 06:33:33.163578 19922 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:33.163645 19922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:33.163801 19922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:33.164878 19922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577196946045-17013-0/minicluster-data/master-0/data/instance:
uuid: "fe6ad218c28a4cd28462f6c8f24ef3fa"
format_stamp: "Formatted at 2026-09-28 06:33:33 on dist-test-slave-rrj5"
server_key: "2e063c1d0be995d4f0bc0f7be73af0e3"
server_key_iv: "bc7d0bb5771e4a6efaacadb9f0aa059d"
server_key_version: "encryptionkey@0"
I20260928 06:33:33.165249 19922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577196946045-17013-0/minicluster-data/master-0/wal/instance:
uuid: "fe6ad218c28a4cd28462f6c8f24ef3fa"
format_stamp: "Formatted at 2026-09-28 06:33:33 on dist-test-slave-rrj5"
server_key: "2e063c1d0be995d4f0bc0f7be73af0e3"
server_key_iv: "bc7d0bb5771e4a6efaacadb9f0aa059d"
server_key_version: "encryptionkey@0"
I20260928 06:33:33.166671 19922 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:33:33.167418 19937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:33.167585 19922 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:33.167685 19922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577196946045-17013-0/minicluster-data/master-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577196946045-17013-0/minicluster-data/master-0/wal
uuid: "fe6ad218c28a4cd28462f6c8f24ef3fa"
format_stamp: "Formatted at 2026-09-28 06:33:33 on dist-test-slave-rrj5"
server_key: "2e063c1d0be995d4f0bc0f7be73af0e3"
server_key_iv: "bc7d0bb5771e4a6efaacadb9f0aa059d"
server_key_version: "encryptionkey@0"
I20260928 06:33:33.167765 19922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577196946045-17013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577196946045-17013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577196946045-17013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:33.176307 19922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:33.178580 19922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:33.178743 19922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:33.183246 19922 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.126:44683
I20260928 06:33:33.183285 19989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.126:44683 every 8 connection(s)
I20260928 06:33:33.183776 19922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
I20260928 06:33:33.184428 19990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:33.187253 19990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe6ad218c28a4cd28462f6c8f24ef3fa: Bootstrap starting.
I20260928 06:33:33.187914 19990 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe6ad218c28a4cd28462f6c8f24ef3fa: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:33.188215 19990 log.cc:824] T 00000000000000000000000000000000 P fe6ad218c28a4cd28462f6c8f24ef3fa: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:33.188963 19990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe6ad218c28a4cd28462f6c8f24ef3fa: No bootstrap required, opened a new log
I20260928 06:33:33.190222 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 19922
I20260928 06:33:33.190315 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577196946045-17013-0/minicluster-data/master-0/wal/instance
I20260928 06:33:33.190492 17013 external_mini_cluster.cc:1428] Setting key 042c163721c3bffeda962551cd10dac9
I20260928 06:33:33.190747 19990 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe6ad218c28a4cd28462f6c8f24ef3fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe6ad218c28a4cd28462f6c8f24ef3fa" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 44683 } }
I20260928 06:33:33.191004 19990 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe6ad218c28a4cd28462f6c8f24ef3fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:33.191084 19990 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe6ad218c28a4cd28462f6c8f24ef3fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe6ad218c28a4cd28462f6c8f24ef3fa, State: Initialized, Role: FOLLOWER
I20260928 06:33:33.191251 19990 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe6ad218c28a4cd28462f6c8f24ef3fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe6ad218c28a4cd28462f6c8f24ef3fa" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 44683 } }
I20260928 06:33:33.191349 19990 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe6ad218c28a4cd28462f6c8f24ef3fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:33:33.191406 19990 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe6ad218c28a4cd28462f6c8f24ef3fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:33:33.191479 19990 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe6ad218c28a4cd28462f6c8f24ef3fa [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:33.192209 19990 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe6ad218c28a4cd28462f6c8f24ef3fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe6ad218c28a4cd28462f6c8f24ef3fa" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 44683 } }
I20260928 06:33:33.192319 19990 leader_election.cc:302] T 00000000000000000000000000000000 P fe6ad218c28a4cd28462f6c8f24ef3fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe6ad218c28a4cd28462f6c8f24ef3fa; no voters:
I20260928 06:33:33.192479 19990 leader_election.cc:288] T 00000000000000000000000000000000 P fe6ad218c28a4cd28462f6c8f24ef3fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:33:33.192507 19995 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe6ad218c28a4cd28462f6c8f24ef3fa [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:33:33.192713 19995 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe6ad218c28a4cd28462f6c8f24ef3fa [term 1 LEADER]: Becoming Leader. State: Replica: fe6ad218c28a4cd28462f6c8f24ef3fa, State: Running, Role: LEADER
I20260928 06:33:33.192816 19990 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe6ad218c28a4cd28462f6c8f24ef3fa [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:33:33.192838 19995 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe6ad218c28a4cd28462f6c8f24ef3fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe6ad218c28a4cd28462f6c8f24ef3fa" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 44683 } }
I20260928 06:33:33.193238 19995 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe6ad218c28a4cd28462f6c8f24ef3fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe6ad218c28a4cd28462f6c8f24ef3fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe6ad218c28a4cd28462f6c8f24ef3fa" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 44683 } } }
I20260928 06:33:33.193410 19995 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe6ad218c28a4cd28462f6c8f24ef3fa [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:33.193638 19995 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe6ad218c28a4cd28462f6c8f24ef3fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe6ad218c28a4cd28462f6c8f24ef3fa. Latest consensus state: current_term: 1 leader_uuid: "fe6ad218c28a4cd28462f6c8f24ef3fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe6ad218c28a4cd28462f6c8f24ef3fa" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 44683 } } }
I20260928 06:33:33.193768 19995 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe6ad218c28a4cd28462f6c8f24ef3fa [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:33.194036 20004 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:33:33.194713 20004 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:33:33.198086 20004 catalog_manager.cc:1347] Generated new cluster ID: 60f6b71afae1466ba97a66772b87e957
I20260928 06:33:33.198153 20004 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:33:33 dist-test-slave-rrj5 krb5kdc[19909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:33:33.206212 19994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:33.191001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53948 (local address 127.16.157.126:44683)
0928 06:33:33.191651 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:33.191661 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:33.191742 (+ 81us) server_negotiation.cc:407] Connection header received
0928 06:33:33.191888 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:33.191895 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:33.191953 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:33.192104 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:33.192924 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:33.194366 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:33.195924 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:33.196541 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:33.196668 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:33.202153 (+ 5485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:33.202157 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:33.202158 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:33.202170 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:33.204232 (+ 2062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:33.204678 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:33.204681 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:33.204681 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:33.204781 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:33.204993 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:33.204993 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:33.204994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:33.205301 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:33.205532 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:33.205838 (+ 306us) server_negotiation.cc:299] Negotiation successful
0928 06:33:33.205996 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":593,"thread_start_us":514,"threads_started":1}
I20260928 06:33:33.207604 17013 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:33:33.212028 20004 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:33:33.212817 20004 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:33:33.218853 20004 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe6ad218c28a4cd28462f6c8f24ef3fa: Generated new TSK 0
I20260928 06:33:33.219111 20004 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:33:33.223702 19994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:33.220286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53954 (local address 127.16.157.126:44683)
0928 06:33:33.220330 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:33.220332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:33.220440 (+ 108us) server_negotiation.cc:407] Connection header received
0928 06:33:33.220514 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:33.220515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:33.220526 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:33.220574 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:33.220967 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:33.221503 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:33.222284 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:33.222496 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:33.222579 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:33.223253 (+ 674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:33.223254 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:33.223254 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:33:33.223447 (+ 193us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:33.223462 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:33:33.223467 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:33.223531 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:33.223593 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 06:33:33.223618 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:33:33.225867 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 19922
2026-09-28T06:33:33Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1379 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:33:33 dist-test-slave-rrj5 krb5kdc[20023](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:33:33 dist-test-slave-rrj5 krb5kdc[20023](info): set up 2 sockets
Sep 28 06:33:33 dist-test-slave-rrj5 krb5kdc[20023](info): commencing operation
krb5kdc: starting...
W20260928 06:33:34.302541 17013 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Sep 28 06:33:34 dist-test-slave-rrj5 krb5kdc[20023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577214, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:33:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:33:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.157.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab.
Entry for principal kudu/127.16.157.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab.
I20260928 06:33:34.424331 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:33173
--webserver_interface=127.16.157.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=kudu/127.16.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:38673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.157.126:33173
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:34.503650 20036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:34.503888 20036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:34.503929 20036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:34.505074 20036 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:33:34.505137 20036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:34.505162 20036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:34.505189 20036 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:33:34.505220 20036 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:33:34.506708 20036 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:38673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.157.126:33173
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=kudu/127.16.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:33173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:34.506975 20036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:34.507201 20036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:34.511926 20041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:34.511972 20042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:34.511972 20044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:34.512351 20036 server_base.cc:1034] running on GCE node
I20260928 06:33:34.512639 20036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:34.512962 20036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:34.514130 20036 hybrid_clock.cc:648] HybridClock initialized: now 1790577214514113 us; error 36 us; skew 500 ppm
Sep 28 06:33:34 dist-test-slave-rrj5 krb5kdc[20023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577214, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:34.516665 20036 init.cc:377] Logged in from keytab as kudu/127.16.157.126@KRBTEST.COM (short username kudu)
I20260928 06:33:34.517141 20036 webserver.cc:460] Webserver started at http://127.16.157.126:42915/ using document root <none> and password file <none>
I20260928 06:33:34.517373 20036 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:34.517441 20036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:34.517602 20036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:34.519019 20036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/minicluster-data/master-0/data/instance:
uuid: "ad2b4bcc893249ad8c4e754c04acaef4"
format_stamp: "Formatted at 2026-09-28 06:33:34 on dist-test-slave-rrj5"
server_key: "3451ffa25ff9c72a897c5ec423d402b5"
server_key_iv: "f1b3b3c975d2f74a1874aa25105c56a2"
server_key_version: "encryptionkey@0"
I20260928 06:33:34.519428 20036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/minicluster-data/master-0/wal/instance:
uuid: "ad2b4bcc893249ad8c4e754c04acaef4"
format_stamp: "Formatted at 2026-09-28 06:33:34 on dist-test-slave-rrj5"
server_key: "3451ffa25ff9c72a897c5ec423d402b5"
server_key_iv: "f1b3b3c975d2f74a1874aa25105c56a2"
server_key_version: "encryptionkey@0"
I20260928 06:33:34.520983 20036 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:33:34.521844 20051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:34.522051 20036 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:33:34.522137 20036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/minicluster-data/master-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/minicluster-data/master-0/wal
uuid: "ad2b4bcc893249ad8c4e754c04acaef4"
format_stamp: "Formatted at 2026-09-28 06:33:34 on dist-test-slave-rrj5"
server_key: "3451ffa25ff9c72a897c5ec423d402b5"
server_key_iv: "f1b3b3c975d2f74a1874aa25105c56a2"
server_key_version: "encryptionkey@0"
I20260928 06:33:34.522202 20036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:34.532675 20036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:34.535200 20036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:34.535353 20036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:34.539570 20036 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.126:33173
I20260928 06:33:34.539614 20103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.126:33173 every 8 connection(s)
I20260928 06:33:34.539968 20036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
I20260928 06:33:34.541118 20104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:34.544062 20104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad2b4bcc893249ad8c4e754c04acaef4: Bootstrap starting.
I20260928 06:33:34.544739 20104 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad2b4bcc893249ad8c4e754c04acaef4: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:34.545044 20104 log.cc:824] T 00000000000000000000000000000000 P ad2b4bcc893249ad8c4e754c04acaef4: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:34.545856 20104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad2b4bcc893249ad8c4e754c04acaef4: No bootstrap required, opened a new log
I20260928 06:33:34.547955 20104 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad2b4bcc893249ad8c4e754c04acaef4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad2b4bcc893249ad8c4e754c04acaef4" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 33173 } }
I20260928 06:33:34.548126 20104 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad2b4bcc893249ad8c4e754c04acaef4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:34.548159 20104 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad2b4bcc893249ad8c4e754c04acaef4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2b4bcc893249ad8c4e754c04acaef4, State: Initialized, Role: FOLLOWER
I20260928 06:33:34.548295 20104 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad2b4bcc893249ad8c4e754c04acaef4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad2b4bcc893249ad8c4e754c04acaef4" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 33173 } }
I20260928 06:33:34.548362 20104 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad2b4bcc893249ad8c4e754c04acaef4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:33:34.548389 20104 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad2b4bcc893249ad8c4e754c04acaef4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:33:34.548432 20104 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad2b4bcc893249ad8c4e754c04acaef4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:34.548676 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 20036
I20260928 06:33:34.548789 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/minicluster-data/master-0/wal/instance
I20260928 06:33:34.548962 17013 external_mini_cluster.cc:1428] Setting key 1e7bd58875d3ed00a35674ee09fe289f
I20260928 06:33:34.549418 20104 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad2b4bcc893249ad8c4e754c04acaef4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad2b4bcc893249ad8c4e754c04acaef4" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 33173 } }
I20260928 06:33:34.549561 20104 leader_election.cc:302] T 00000000000000000000000000000000 P ad2b4bcc893249ad8c4e754c04acaef4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad2b4bcc893249ad8c4e754c04acaef4; no voters:
I20260928 06:33:34.549760 20104 leader_election.cc:288] T 00000000000000000000000000000000 P ad2b4bcc893249ad8c4e754c04acaef4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:33:34.549789 20108 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad2b4bcc893249ad8c4e754c04acaef4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:33:34.549974 20108 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad2b4bcc893249ad8c4e754c04acaef4 [term 1 LEADER]: Becoming Leader. State: Replica: ad2b4bcc893249ad8c4e754c04acaef4, State: Running, Role: LEADER
I20260928 06:33:34.550016 20104 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad2b4bcc893249ad8c4e754c04acaef4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:33:34.550127 20108 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad2b4bcc893249ad8c4e754c04acaef4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad2b4bcc893249ad8c4e754c04acaef4" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 33173 } }
I20260928 06:33:34.550607 20111 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad2b4bcc893249ad8c4e754c04acaef4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad2b4bcc893249ad8c4e754c04acaef4. Latest consensus state: current_term: 1 leader_uuid: "ad2b4bcc893249ad8c4e754c04acaef4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad2b4bcc893249ad8c4e754c04acaef4" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 33173 } } }
I20260928 06:33:34.550628 20110 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad2b4bcc893249ad8c4e754c04acaef4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad2b4bcc893249ad8c4e754c04acaef4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad2b4bcc893249ad8c4e754c04acaef4" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 33173 } } }
I20260928 06:33:34.550779 20111 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad2b4bcc893249ad8c4e754c04acaef4 [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:34.550799 20110 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad2b4bcc893249ad8c4e754c04acaef4 [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:34.551206 20118 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:33:34.551741 20118 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:33:34.553756 20118 catalog_manager.cc:1347] Generated new cluster ID: fbf7374a7b14484a9770bf548d25e586
I20260928 06:33:34.553853 20118 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:33:34 dist-test-slave-rrj5 krb5kdc[20023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577214, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:33:34.562654 20109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:34.549643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33584 (local address 127.16.157.126:33173)
0928 06:33:34.549914 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:34.549925 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:34.549961 (+ 36us) server_negotiation.cc:407] Connection header received
0928 06:33:34.550084 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:34.550092 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:34.550173 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:34.551201 (+ 1028us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:34.551310 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:34.552865 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:34.553995 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:34.554599 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:34.554968 (+ 369us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:34.557935 (+ 2967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:34.557939 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:34.557940 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:34.557953 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:34.560665 (+ 2712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:34.561039 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:34.561042 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:34.561042 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:34.561158 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:34.561325 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:34.561326 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:34.561326 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:34.561655 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:34.561787 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:34.562257 (+ 470us) server_negotiation.cc:299] Negotiation successful
0928 06:33:34.562393 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":111,"threads_started":1}
I20260928 06:33:34.564808 20118 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:33:34.565303 20118 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:33:34.568717 20118 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad2b4bcc893249ad8c4e754c04acaef4: Generated new TSK 0
I20260928 06:33:34.568910 20118 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.157.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab.
Entry for principal kudu/127.16.157.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab.
I20260928 06:33:34.596202 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.65:0
--local_ip_for_outbound_sockets=127.16.157.65
--webserver_interface=127.16.157.65
--webserver_port=0
--tserver_master_addrs=127.16.157.126:33173
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab
--principal=kudu/127.16.157.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:38673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:34.679421 20130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:34.679683 20130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:34.679726 20130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:34.681035 20130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:34.681103 20130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:34.681149 20130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.65
I20260928 06:33:34.682726 20130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:38673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab
--principal=kudu/127.16.157.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.157.65
--webserver_port=0
--tserver_master_addrs=127.16.157.126:33173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.65
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:34.683013 20130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:34.683264 20130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:34.688457 20135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:34.688803 20138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:34.688859 20136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:34.689344 20130 server_base.cc:1034] running on GCE node
I20260928 06:33:34.689623 20130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:34.689929 20130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:34.691088 20130 hybrid_clock.cc:648] HybridClock initialized: now 1790577214691073 us; error 40 us; skew 500 ppm
Sep 28 06:33:34 dist-test-slave-rrj5 krb5kdc[20023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577214, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:34.693804 20130 init.cc:377] Logged in from keytab as kudu/127.16.157.65@KRBTEST.COM (short username kudu)
I20260928 06:33:34.694283 20130 webserver.cc:460] Webserver started at http://127.16.157.65:42605/ using document root <none> and password file <none>
I20260928 06:33:34.694521 20130 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:34.694607 20130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:34.694748 20130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:34.695847 20130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-0/data/instance:
uuid: "b17aa7a4be0b47089926fe470b097360"
format_stamp: "Formatted at 2026-09-28 06:33:34 on dist-test-slave-rrj5"
server_key: "7e262c1184b63cc01106873623684a7d"
server_key_iv: "2b39de246a341360638309d8a1218f27"
server_key_version: "encryptionkey@0"
I20260928 06:33:34.696231 20130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-0/wal/instance:
uuid: "b17aa7a4be0b47089926fe470b097360"
format_stamp: "Formatted at 2026-09-28 06:33:34 on dist-test-slave-rrj5"
server_key: "7e262c1184b63cc01106873623684a7d"
server_key_iv: "2b39de246a341360638309d8a1218f27"
server_key_version: "encryptionkey@0"
I20260928 06:33:34.697727 20130 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:33:34.698565 20145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:34.698721 20130 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:34.698807 20130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-0/wal
uuid: "b17aa7a4be0b47089926fe470b097360"
format_stamp: "Formatted at 2026-09-28 06:33:34 on dist-test-slave-rrj5"
server_key: "7e262c1184b63cc01106873623684a7d"
server_key_iv: "2b39de246a341360638309d8a1218f27"
server_key_version: "encryptionkey@0"
I20260928 06:33:34.698874 20130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:34.714727 20130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:34.717140 20130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:34.717309 20130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:34.717628 20130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:33:34.718016 20130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:33:34.718070 20130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:34.718127 20130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:33:34.718158 20130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:34.725518 20130 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.65:38111
I20260928 06:33:34.725584 20258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.65:38111 every 8 connection(s)
I20260928 06:33:34.726042 20130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
I20260928 06:33:34.731384 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 20130
I20260928 06:33:34.731493 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-0/wal/instance
I20260928 06:33:34.731670 17013 external_mini_cluster.cc:1428] Setting key 540c063bae9c16ea3b2cad1c09426057
Sep 28 06:33:34 dist-test-slave-rrj5 krb5kdc[20023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577214, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.65@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:33:34.736459 20261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:34.726769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:33173 (local address 127.16.157.65:44445)
0928 06:33:34.726920 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:34.726961 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:34.727486 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:34.727735 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:34.727739 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:34.728298 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:34.728488 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:34.728493 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:34.729201 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:34.729202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:34.730115 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:34.730117 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:34.730544 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:34.730545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:34.730677 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:34.731206 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:34.731215 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:34.732917 (+ 1702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:34.734891 (+ 1974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:34.734898 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:34.734900 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:34.735193 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:34.735417 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:34.735418 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:34.735418 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:34.735545 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:34.735824 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:34.735829 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:34.735979 (+ 150us) client_negotiation.cc:769] Sending connection context
0928 06:33:34.736082 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 06:33:34.736239 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":79,"threads_started":1}
I20260928 06:33:34.737164 20109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:34.726761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:44445 (local address 127.16.157.126:33173)
0928 06:33:34.726992 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:34.726994 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:34.727054 (+ 60us) server_negotiation.cc:407] Connection header received
0928 06:33:34.727600 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:34.727602 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:34.727616 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:34.727667 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:34.728589 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:34.729101 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:34.730207 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:34.730435 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:34.730530 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:34.733021 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:34.733023 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:34.733023 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:34.733032 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:34.734824 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:34.735289 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:34.735290 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:34.735291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:34.735352 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:34.735614 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:34.735615 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:34.735615 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:34.735743 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:34.735799 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:34.736984 (+ 1185us) server_negotiation.cc:299] Negotiation successful
0928 06:33:34.737072 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207}
I20260928 06:33:34.737860 20259 heartbeater.cc:347] Connected to a master server at 127.16.157.126:33173
I20260928 06:33:34.737977 20259 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:34.738181 20259 heartbeater.cc:511] Master 127.16.157.126:33173 requested a full tablet report, sending...
I20260928 06:33:34.738790 20068 ts_manager.cc:194] Registered new tserver with Master: b17aa7a4be0b47089926fe470b097360 (127.16.157.65:38111)
I20260928 06:33:34.739670 20068 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.157.65@KRBTEST.COM'} at 127.16.157.65:44445
WARNING: no policy specified for kudu/127.16.157.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab.
Entry for principal kudu/127.16.157.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab.
I20260928 06:33:34.757441 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.66:0
--local_ip_for_outbound_sockets=127.16.157.66
--webserver_interface=127.16.157.66
--webserver_port=0
--tserver_master_addrs=127.16.157.126:33173
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab
--principal=kudu/127.16.157.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:38673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:34.838775 20264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:34.838963 20264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:34.838986 20264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:34.840042 20264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:34.840076 20264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:34.840102 20264 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.66
I20260928 06:33:34.841555 20264 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:38673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab
--principal=kudu/127.16.157.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.157.66
--webserver_port=0
--tserver_master_addrs=127.16.157.126:33173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20264
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.66
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:34.841763 20264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:34.841995 20264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:34.848094 20270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:34.848030 20269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:34.848300 20264 server_base.cc:1034] running on GCE node
W20260928 06:33:34.848316 20272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:34.848618 20264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:34.848892 20264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:34.850061 20264 hybrid_clock.cc:648] HybridClock initialized: now 1790577214850038 us; error 39 us; skew 500 ppm
Sep 28 06:33:34 dist-test-slave-rrj5 krb5kdc[20023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577214, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:34.857352 20264 init.cc:377] Logged in from keytab as kudu/127.16.157.66@KRBTEST.COM (short username kudu)
I20260928 06:33:34.857909 20264 webserver.cc:460] Webserver started at http://127.16.157.66:39781/ using document root <none> and password file <none>
I20260928 06:33:34.858175 20264 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:34.858225 20264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:34.858407 20264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:34.859476 20264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-1/data/instance:
uuid: "308be77108ce40a39ac19449000658df"
format_stamp: "Formatted at 2026-09-28 06:33:34 on dist-test-slave-rrj5"
server_key: "fbb4e4728d0bbc6ca0f57a6464ca97d3"
server_key_iv: "9fdf055f095140b20349e64b14007210"
server_key_version: "encryptionkey@0"
I20260928 06:33:34.859845 20264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-1/wal/instance:
uuid: "308be77108ce40a39ac19449000658df"
format_stamp: "Formatted at 2026-09-28 06:33:34 on dist-test-slave-rrj5"
server_key: "fbb4e4728d0bbc6ca0f57a6464ca97d3"
server_key_iv: "9fdf055f095140b20349e64b14007210"
server_key_version: "encryptionkey@0"
I20260928 06:33:34.861260 20264 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:33:34.862219 20279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:34.862433 20264 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:33:34.862540 20264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-1/wal
uuid: "308be77108ce40a39ac19449000658df"
format_stamp: "Formatted at 2026-09-28 06:33:34 on dist-test-slave-rrj5"
server_key: "fbb4e4728d0bbc6ca0f57a6464ca97d3"
server_key_iv: "9fdf055f095140b20349e64b14007210"
server_key_version: "encryptionkey@0"
I20260928 06:33:34.862634 20264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:34.885363 20264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:34.887878 20264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:34.888068 20264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:34.888329 20264 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:33:34.888670 20264 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:33:34.888731 20264 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:34.888805 20264 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:33:34.888839 20264 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:34.895606 20264 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.66:34663
I20260928 06:33:34.895658 20392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.66:34663 every 8 connection(s)
I20260928 06:33:34.896029 20264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
I20260928 06:33:34.902498 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 20264
I20260928 06:33:34.902638 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-1/wal/instance
I20260928 06:33:34.902855 17013 external_mini_cluster.cc:1428] Setting key d19ece58a72196468adf504e4ee0bdf9
Sep 28 06:33:34 dist-test-slave-rrj5 krb5kdc[20023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577214, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.66@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:33:34.908001 20109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:34.897037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.66:57493 (local address 127.16.157.126:33173)
0928 06:33:34.897138 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:34.897140 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:34.897179 (+ 39us) server_negotiation.cc:407] Connection header received
0928 06:33:34.897982 (+ 803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:34.897983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:34.897994 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:34.898051 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:34.899365 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:34.899863 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:34.901532 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:34.901808 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:34.901912 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:34.904560 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:34.904562 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:34.904562 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:34.904571 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:34.906337 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:34.906805 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:34.906806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:34.906806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:34.906876 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:34.907211 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:34.907212 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:34.907212 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:34.907358 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:34.907439 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:34.907789 (+ 350us) server_negotiation.cc:299] Negotiation successful
0928 06:33:34.907883 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 06:33:34.908250 20395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:34.896808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:33173 (local address 127.16.157.66:57493)
0928 06:33:34.897000 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:34.897070 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:34.897841 (+ 771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:34.898145 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:34.898149 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:34.898931 (+ 782us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:34.899238 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:34.899243 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:34.899960 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:34.899961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:34.901418 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:34.901421 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:34.901939 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:34.901941 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:34.902086 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:34.902624 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:34.902635 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:34.904455 (+ 1820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:34.906450 (+ 1995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:34.906458 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:34.906461 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:34.906701 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:34.906965 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:34.906966 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:34.906966 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:34.907114 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:34.907445 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:34.907453 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:34.907624 (+ 171us) client_negotiation.cc:769] Sending connection context
0928 06:33:34.907753 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 06:33:34.907940 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":94,"threads_started":1}
I20260928 06:33:34.908954 20393 heartbeater.cc:347] Connected to a master server at 127.16.157.126:33173
I20260928 06:33:34.909070 20393 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:34.909266 20393 heartbeater.cc:511] Master 127.16.157.126:33173 requested a full tablet report, sending...
I20260928 06:33:34.909711 20068 ts_manager.cc:194] Registered new tserver with Master: 308be77108ce40a39ac19449000658df (127.16.157.66:34663)
I20260928 06:33:34.910102 20068 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.157.66@KRBTEST.COM'} at 127.16.157.66:57493
WARNING: no policy specified for kudu/127.16.157.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab.
Entry for principal kudu/127.16.157.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab.
I20260928 06:33:34.929981 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.67:0
--local_ip_for_outbound_sockets=127.16.157.67
--webserver_interface=127.16.157.67
--webserver_port=0
--tserver_master_addrs=127.16.157.126:33173
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab
--principal=kudu/127.16.157.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:38673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:35.012801 20398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:35.013044 20398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:35.013084 20398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:35.014309 20398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:35.014374 20398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:35.014420 20398 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.67
I20260928 06:33:35.015916 20398 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:38673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab
--principal=kudu/127.16.157.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.157.67
--webserver_port=0
--tserver_master_addrs=127.16.157.126:33173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20398
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.67
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:35.016184 20398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:35.016435 20398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:35.021662 20404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:35.021646 20406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:35.021911 20403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:35.021966 20398 server_base.cc:1034] running on GCE node
I20260928 06:33:35.022163 20398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:35.022452 20398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:35.023654 20398 hybrid_clock.cc:648] HybridClock initialized: now 1790577215023635 us; error 37 us; skew 500 ppm
Sep 28 06:33:35 dist-test-slave-rrj5 krb5kdc[20023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577215, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:35.026427 20398 init.cc:377] Logged in from keytab as kudu/127.16.157.67@KRBTEST.COM (short username kudu)
I20260928 06:33:35.026988 20398 webserver.cc:460] Webserver started at http://127.16.157.67:45309/ using document root <none> and password file <none>
I20260928 06:33:35.027240 20398 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:35.027309 20398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:35.027455 20398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:35.028504 20398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-2/data/instance:
uuid: "a3534a8e904441dbbad6462582a63f54"
format_stamp: "Formatted at 2026-09-28 06:33:35 on dist-test-slave-rrj5"
server_key: "d46497be619d7ee3a33982eda1e6adc9"
server_key_iv: "d6177a784ba73272e5ff1e56a9762ad2"
server_key_version: "encryptionkey@0"
I20260928 06:33:35.028859 20398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-2/wal/instance:
uuid: "a3534a8e904441dbbad6462582a63f54"
format_stamp: "Formatted at 2026-09-28 06:33:35 on dist-test-slave-rrj5"
server_key: "d46497be619d7ee3a33982eda1e6adc9"
server_key_iv: "d6177a784ba73272e5ff1e56a9762ad2"
server_key_version: "encryptionkey@0"
I20260928 06:33:35.030318 20398 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:33:35.031169 20413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:35.031366 20398 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:35.031467 20398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-2/wal
uuid: "a3534a8e904441dbbad6462582a63f54"
format_stamp: "Formatted at 2026-09-28 06:33:35 on dist-test-slave-rrj5"
server_key: "d46497be619d7ee3a33982eda1e6adc9"
server_key_iv: "d6177a784ba73272e5ff1e56a9762ad2"
server_key_version: "encryptionkey@0"
I20260928 06:33:35.031553 20398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:35.076460 20398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:35.078755 20398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:35.078927 20398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:35.079180 20398 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:33:35.079491 20398 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:33:35.079550 20398 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:35.079622 20398 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:33:35.079650 20398 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:35.086773 20398 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.67:43249
I20260928 06:33:35.086807 20526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.67:43249 every 8 connection(s)
I20260928 06:33:35.087160 20398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
Sep 28 06:33:35 dist-test-slave-rrj5 krb5kdc[20023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577215, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.67@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:33:35.095474 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 20398
I20260928 06:33:35.095578 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577196946045-17013-0/minicluster-data/ts-2/wal/instance
I20260928 06:33:35.095773 17013 external_mini_cluster.cc:1428] Setting key fe4ebd944bb754c98913a8c78bcc87e3
I20260928 06:33:35.098279 20109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:35.087942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:59831 (local address 127.16.157.126:33173)
0928 06:33:35.087993 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:35.087995 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:35.088379 (+ 384us) server_negotiation.cc:407] Connection header received
0928 06:33:35.089015 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:35.089016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:35.089025 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:35.089094 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:35.090422 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:35.090927 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:35.092330 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:35.092511 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:35.092561 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:35.094960 (+ 2399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:35.094962 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:35.094962 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:35.094971 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:35.096826 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:35.097274 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:35.097275 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:35.097275 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:35.097327 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:35.097644 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:35.097645 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:35.097645 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:35.097764 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:35.097813 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:35.098150 (+ 337us) server_negotiation.cc:299] Negotiation successful
0928 06:33:35.098217 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:33:35.098518 20529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:35.087935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:33173 (local address 127.16.157.67:59831)
0928 06:33:35.088221 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:35.088273 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:35.088908 (+ 635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:35.089176 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:35.089181 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:35.090030 (+ 849us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:35.090326 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:35.090333 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:35.091039 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:35.091040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:35.092244 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:35.092247 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:35.092596 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:35.092597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:35.092754 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:35.093289 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:35.093300 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:35.094866 (+ 1566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:35.096902 (+ 2036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:35.096909 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:35.096912 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:35.097201 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:35.097392 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:35.097393 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:35.097393 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:35.097541 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:35.097829 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:35.097836 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:35.098001 (+ 165us) client_negotiation.cc:769] Sending connection context
0928 06:33:35.098116 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 06:33:35.098283 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":212,"threads_started":1}
I20260928 06:33:35.099061 20527 heartbeater.cc:347] Connected to a master server at 127.16.157.126:33173
I20260928 06:33:35.099179 20527 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:35.099356 20527 heartbeater.cc:511] Master 127.16.157.126:33173 requested a full tablet report, sending...
I20260928 06:33:35.099745 20068 ts_manager.cc:194] Registered new tserver with Master: a3534a8e904441dbbad6462582a63f54 (127.16.157.67:43249)
I20260928 06:33:35.100127 20068 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.157.67@KRBTEST.COM'} at 127.16.157.67:59831
I20260928 06:33:35.107877 17013 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 06:33:35 dist-test-slave-rrj5 krb5kdc[20023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577215, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 06:33:35 dist-test-slave-rrj5 krb5kdc[20023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577215, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:33:35.131500 20109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:35.123611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33586 (local address 127.16.157.126:33173)
0928 06:33:35.123681 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:35.123683 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:35.123804 (+ 121us) server_negotiation.cc:407] Connection header received
0928 06:33:35.123874 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:35.123875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:35.123884 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:35.123936 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:35.124605 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:35.125093 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:35.126184 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:35.126372 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:35.126422 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:35.128909 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:35.128911 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:35.128911 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:35.128919 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:35.130299 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:35.130657 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:35.130658 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:35.130659 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:35.130718 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:35.130929 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:35.130930 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:35.130930 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:35.131088 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:35.131194 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:35.131305 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 06:33:35.131387 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260928 06:33:35.131713 20068 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:33586
I20260928 06:33:35.132107 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 20130
I20260928 06:33:35.137403 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 20264
I20260928 06:33:35.143594 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 20398
I20260928 06:33:35.148101 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 20036
2026-09-28T06:33:35Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1925 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:33:35 dist-test-slave-rrj5 krb5kdc[20539](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:33:35 dist-test-slave-rrj5 krb5kdc[20539](info): set up 2 sockets
Sep 28 06:33:35 dist-test-slave-rrj5 krb5kdc[20539](info): commencing operation
krb5kdc: starting...
W20260928 06:33:36.239406 17013 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Sep 28 06:33:36 dist-test-slave-rrj5 krb5kdc[20539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:33:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:33:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.157.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab.
Entry for principal kudu/127.16.157.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab.
I20260928 06:33:36.411917 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:43035
--webserver_interface=127.16.157.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=kudu/127.16.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:38785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.157.126:43035
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:36.499830 20552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:36.500068 20552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:36.500105 20552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:36.501224 20552 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:33:36.501284 20552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:36.501308 20552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:36.501339 20552 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:33:36.501364 20552 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:33:36.502904 20552 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:38785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.157.126:43035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=kudu/127.16.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:43035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:36.503170 20552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:36.503438 20552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:36.508407 20558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:36.508441 20560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:36.508561 20557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:36.508566 20552 server_base.cc:1034] running on GCE node
I20260928 06:33:36.508857 20552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:36.509171 20552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:36.510365 20552 hybrid_clock.cc:648] HybridClock initialized: now 1790577216510350 us; error 29 us; skew 500 ppm
Sep 28 06:33:36 dist-test-slave-rrj5 krb5kdc[20539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577216, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:36.512967 20552 init.cc:377] Logged in from keytab as kudu/127.16.157.126@KRBTEST.COM (short username kudu)
I20260928 06:33:36.513427 20552 webserver.cc:460] Webserver started at http://127.16.157.126:43859/ using document root <none> and password file <none>
I20260928 06:33:36.513715 20552 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:36.513783 20552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:36.513918 20552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:36.514932 20552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/minicluster-data/master-0/data/instance:
uuid: "cabfc9e268fb4cf7bc74780b1e618ce0"
format_stamp: "Formatted at 2026-09-28 06:33:36 on dist-test-slave-rrj5"
server_key: "b28fecae6a5b2943422c454982fc51d7"
server_key_iv: "10e5276548406fdde75cf14f766574c4"
server_key_version: "encryptionkey@0"
I20260928 06:33:36.515288 20552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/minicluster-data/master-0/wal/instance:
uuid: "cabfc9e268fb4cf7bc74780b1e618ce0"
format_stamp: "Formatted at 2026-09-28 06:33:36 on dist-test-slave-rrj5"
server_key: "b28fecae6a5b2943422c454982fc51d7"
server_key_iv: "10e5276548406fdde75cf14f766574c4"
server_key_version: "encryptionkey@0"
I20260928 06:33:36.516592 20552 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:33:36.517468 20567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:36.517655 20552 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:36.517753 20552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/minicluster-data/master-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/minicluster-data/master-0/wal
uuid: "cabfc9e268fb4cf7bc74780b1e618ce0"
format_stamp: "Formatted at 2026-09-28 06:33:36 on dist-test-slave-rrj5"
server_key: "b28fecae6a5b2943422c454982fc51d7"
server_key_iv: "10e5276548406fdde75cf14f766574c4"
server_key_version: "encryptionkey@0"
I20260928 06:33:36.517835 20552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:36.537313 20552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:36.539971 20552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:36.540207 20552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:36.544986 20552 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.126:43035
I20260928 06:33:36.545064 20619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.126:43035 every 8 connection(s)
I20260928 06:33:36.545631 20552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
I20260928 06:33:36.546325 20620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:36.547247 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 20552
I20260928 06:33:36.547361 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/minicluster-data/master-0/wal/instance
I20260928 06:33:36.547546 17013 external_mini_cluster.cc:1428] Setting key 98a5c6844071036968066f63a8d67bfd
I20260928 06:33:36.549903 20620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cabfc9e268fb4cf7bc74780b1e618ce0: Bootstrap starting.
I20260928 06:33:36.550524 20620 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cabfc9e268fb4cf7bc74780b1e618ce0: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:36.550865 20620 log.cc:824] T 00000000000000000000000000000000 P cabfc9e268fb4cf7bc74780b1e618ce0: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:36.551667 20620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cabfc9e268fb4cf7bc74780b1e618ce0: No bootstrap required, opened a new log
I20260928 06:33:36.553978 20620 raft_consensus.cc:348] T 00000000000000000000000000000000 P cabfc9e268fb4cf7bc74780b1e618ce0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cabfc9e268fb4cf7bc74780b1e618ce0" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 43035 } }
I20260928 06:33:36.554138 20620 raft_consensus.cc:374] T 00000000000000000000000000000000 P cabfc9e268fb4cf7bc74780b1e618ce0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:36.554184 20620 raft_consensus.cc:729] T 00000000000000000000000000000000 P cabfc9e268fb4cf7bc74780b1e618ce0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cabfc9e268fb4cf7bc74780b1e618ce0, State: Initialized, Role: FOLLOWER
I20260928 06:33:36.554369 20620 consensus_queue.cc:260] T 00000000000000000000000000000000 P cabfc9e268fb4cf7bc74780b1e618ce0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cabfc9e268fb4cf7bc74780b1e618ce0" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 43035 } }
I20260928 06:33:36.554484 20620 raft_consensus.cc:388] T 00000000000000000000000000000000 P cabfc9e268fb4cf7bc74780b1e618ce0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:33:36.554536 20620 raft_consensus.cc:482] T 00000000000000000000000000000000 P cabfc9e268fb4cf7bc74780b1e618ce0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:33:36.554584 20620 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cabfc9e268fb4cf7bc74780b1e618ce0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:36.555176 20620 raft_consensus.cc:504] T 00000000000000000000000000000000 P cabfc9e268fb4cf7bc74780b1e618ce0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cabfc9e268fb4cf7bc74780b1e618ce0" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 43035 } }
I20260928 06:33:36.555279 20620 leader_election.cc:302] T 00000000000000000000000000000000 P cabfc9e268fb4cf7bc74780b1e618ce0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cabfc9e268fb4cf7bc74780b1e618ce0; no voters:
I20260928 06:33:36.555490 20620 leader_election.cc:288] T 00000000000000000000000000000000 P cabfc9e268fb4cf7bc74780b1e618ce0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:33:36.555562 20625 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cabfc9e268fb4cf7bc74780b1e618ce0 [term 1 FOLLOWER]: Leader election won for term 1
Sep 28 06:33:36 dist-test-slave-rrj5 krb5kdc[20539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:33:36.555752 20620 sys_catalog.cc:564] T 00000000000000000000000000000000 P cabfc9e268fb4cf7bc74780b1e618ce0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:33:36.555757 20625 raft_consensus.cc:686] T 00000000000000000000000000000000 P cabfc9e268fb4cf7bc74780b1e618ce0 [term 1 LEADER]: Becoming Leader. State: Replica: cabfc9e268fb4cf7bc74780b1e618ce0, State: Running, Role: LEADER
I20260928 06:33:36.555924 20625 consensus_queue.cc:237] T 00000000000000000000000000000000 P cabfc9e268fb4cf7bc74780b1e618ce0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cabfc9e268fb4cf7bc74780b1e618ce0" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 43035 } }
I20260928 06:33:36.556720 20627 sys_catalog.cc:455] T 00000000000000000000000000000000 P cabfc9e268fb4cf7bc74780b1e618ce0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cabfc9e268fb4cf7bc74780b1e618ce0. Latest consensus state: current_term: 1 leader_uuid: "cabfc9e268fb4cf7bc74780b1e618ce0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cabfc9e268fb4cf7bc74780b1e618ce0" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 43035 } } }
I20260928 06:33:36.556810 20627 sys_catalog.cc:458] T 00000000000000000000000000000000 P cabfc9e268fb4cf7bc74780b1e618ce0 [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:36.556915 20626 sys_catalog.cc:455] T 00000000000000000000000000000000 P cabfc9e268fb4cf7bc74780b1e618ce0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cabfc9e268fb4cf7bc74780b1e618ce0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cabfc9e268fb4cf7bc74780b1e618ce0" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 43035 } } }
I20260928 06:33:36.556975 20626 sys_catalog.cc:458] T 00000000000000000000000000000000 P cabfc9e268fb4cf7bc74780b1e618ce0 [sys.catalog]: This master's current role is: LEADER
W20260928 06:33:36.557847 20640 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cabfc9e268fb4cf7bc74780b1e618ce0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:33:36.557961 20640 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:33:36.558063 20636 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:33:36.558244 20636 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:33:36.560057 20636 catalog_manager.cc:1347] Generated new cluster ID: 45389d52a9bc46e488ad4b52f1c628cc
I20260928 06:33:36.560107 20636 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:33:36.560539 20623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:36.548064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35696 (local address 127.16.157.126:43035)
0928 06:33:36.548255 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:36.548265 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:36.548336 (+ 71us) server_negotiation.cc:407] Connection header received
0928 06:33:36.548942 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:36.548950 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:36.549013 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:36.549157 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:36.549972 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:36.551449 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:36.552504 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:36.553034 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:36.553166 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:36.556170 (+ 3004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:36.556174 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:36.556174 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:36.556185 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:36.558852 (+ 2667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:36.559273 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:36.559276 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:36.559277 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:36.559388 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:36.559602 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:36.559603 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:36.559603 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:36.559915 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:36.560097 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:36.560254 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 06:33:36.560391 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":78,"threads_started":1}
I20260928 06:33:36.588073 20636 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:33:36.588805 20636 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:33:36.593098 20636 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cabfc9e268fb4cf7bc74780b1e618ce0: Generated new TSK 0
I20260928 06:33:36.593271 20636 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.157.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab.
Entry for principal kudu/127.16.157.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab.
I20260928 06:33:36.628466 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.65:0
--local_ip_for_outbound_sockets=127.16.157.65
--webserver_interface=127.16.157.65
--webserver_port=0
--tserver_master_addrs=127.16.157.126:43035
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab
--principal=kudu/127.16.157.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:38785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:36.737057 20646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:36.737311 20646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:36.737339 20646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:36.738672 20646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:36.738714 20646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:36.738741 20646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.65
I20260928 06:33:36.741241 20646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:38785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab
--principal=kudu/127.16.157.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.157.65
--webserver_port=0
--tserver_master_addrs=127.16.157.126:43035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.65
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:36.741485 20646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:36.741772 20646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:36.747285 20652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:36.747380 20651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:36.747542 20646 server_base.cc:1034] running on GCE node
W20260928 06:33:36.747288 20654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:36.747756 20646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:36.748008 20646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:36.749174 20646 hybrid_clock.cc:648] HybridClock initialized: now 1790577216749166 us; error 35 us; skew 500 ppm
Sep 28 06:33:36 dist-test-slave-rrj5 krb5kdc[20539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577216, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:36.752044 20646 init.cc:377] Logged in from keytab as kudu/127.16.157.65@KRBTEST.COM (short username kudu)
I20260928 06:33:36.752508 20646 webserver.cc:460] Webserver started at http://127.16.157.65:39175/ using document root <none> and password file <none>
I20260928 06:33:36.752758 20646 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:36.752802 20646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:36.752930 20646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:36.754398 20646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/minicluster-data/ts-0/data/instance:
uuid: "2240def25b1a4f39b4b3d828d1fe4a79"
format_stamp: "Formatted at 2026-09-28 06:33:36 on dist-test-slave-rrj5"
server_key: "a25e604e11fd07e8883344609977b58b"
server_key_iv: "d3e68dc605ced604ef2fab12a6a2a336"
server_key_version: "encryptionkey@0"
I20260928 06:33:36.754851 20646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/minicluster-data/ts-0/wal/instance:
uuid: "2240def25b1a4f39b4b3d828d1fe4a79"
format_stamp: "Formatted at 2026-09-28 06:33:36 on dist-test-slave-rrj5"
server_key: "a25e604e11fd07e8883344609977b58b"
server_key_iv: "d3e68dc605ced604ef2fab12a6a2a336"
server_key_version: "encryptionkey@0"
I20260928 06:33:36.756314 20646 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:33:36.757100 20661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:36.757298 20646 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:36.757404 20646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/minicluster-data/ts-0/wal
uuid: "2240def25b1a4f39b4b3d828d1fe4a79"
format_stamp: "Formatted at 2026-09-28 06:33:36 on dist-test-slave-rrj5"
server_key: "a25e604e11fd07e8883344609977b58b"
server_key_iv: "d3e68dc605ced604ef2fab12a6a2a336"
server_key_version: "encryptionkey@0"
I20260928 06:33:36.757499 20646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:36.775365 20646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:36.777523 20646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:36.777740 20646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:36.777994 20646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:33:36.778309 20646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:33:36.778364 20646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:36.778407 20646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:33:36.778432 20646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:36.784476 20646 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.65:34085
I20260928 06:33:36.784556 20774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.65:34085 every 8 connection(s)
I20260928 06:33:36.784866 20646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
Sep 28 06:33:36 dist-test-slave-rrj5 krb5kdc[20539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577216, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.65@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:33:36.794373 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 20646
I20260928 06:33:36.794499 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/minicluster-data/ts-0/wal/instance
I20260928 06:33:36.794723 17013 external_mini_cluster.cc:1428] Setting key 88744a643bd72dc2a2196e4ab35d9fa1
I20260928 06:33:36.795017 20623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:36.785713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:56481 (local address 127.16.157.126:43035)
0928 06:33:36.785891 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:36.785893 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:36.786033 (+ 140us) server_negotiation.cc:407] Connection header received
0928 06:33:36.787022 (+ 989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:36.787024 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:36.787036 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:36.787090 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:36.787785 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:36.788340 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:36.789508 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:36.789783 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:36.789842 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:36.792066 (+ 2224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:36.792068 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:36.792068 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:36.792074 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:36.793347 (+ 1273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:36.793855 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:36.793856 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:36.793856 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:36.793909 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:36.794188 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:36.794189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:36.794190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:36.794344 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:36.794422 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:36.794808 (+ 386us) server_negotiation.cc:299] Negotiation successful
0928 06:33:36.794896 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 06:33:36.795178 20777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:36.785707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:43035 (local address 127.16.157.65:56481)
0928 06:33:36.785894 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:36.785936 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:36.786894 (+ 958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:36.787183 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:36.787187 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:36.787496 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:36.787694 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:36.787699 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:36.788422 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:36.788422 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:36.789404 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:36.789413 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:36.789859 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:36.789860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:36.789961 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:36.790453 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:36.790463 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:36.791976 (+ 1513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:36.793426 (+ 1450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:36.793433 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:36.793435 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:36.793777 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:36.793974 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:36.793975 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:36.793975 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:36.794110 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:36.794422 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:36.794445 (+ 23us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:36.794646 (+ 201us) client_negotiation.cc:769] Sending connection context
0928 06:33:36.794770 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 06:33:36.794942 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":85,"threads_started":1}
I20260928 06:33:36.795646 20775 heartbeater.cc:347] Connected to a master server at 127.16.157.126:43035
I20260928 06:33:36.795734 20775 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:36.796005 20775 heartbeater.cc:511] Master 127.16.157.126:43035 requested a full tablet report, sending...
I20260928 06:33:36.797366 20584 ts_manager.cc:194] Registered new tserver with Master: 2240def25b1a4f39b4b3d828d1fe4a79 (127.16.157.65:34085)
I20260928 06:33:36.798053 20584 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.157.65@KRBTEST.COM'} at 127.16.157.65:56481
WARNING: no policy specified for kudu/127.16.157.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab.
Entry for principal kudu/127.16.157.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab.
I20260928 06:33:36.819901 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.66:0
--local_ip_for_outbound_sockets=127.16.157.66
--webserver_interface=127.16.157.66
--webserver_port=0
--tserver_master_addrs=127.16.157.126:43035
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab
--principal=kudu/127.16.157.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:38785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:36.932204 20780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:36.932475 20780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:36.932513 20780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:36.934208 20780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:36.934260 20780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:36.934297 20780 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.66
I20260928 06:33:36.936460 20780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:38785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab
--principal=kudu/127.16.157.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.157.66
--webserver_port=0
--tserver_master_addrs=127.16.157.126:43035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.66
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:36.936685 20780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:36.936924 20780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:36.942292 20786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:36.942564 20780 server_base.cc:1034] running on GCE node
W20260928 06:33:36.942696 20788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:36.945261 20785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:36.945540 20780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:36.945878 20780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:36.947047 20780 hybrid_clock.cc:648] HybridClock initialized: now 1790577216947027 us; error 34 us; skew 500 ppm
Sep 28 06:33:36 dist-test-slave-rrj5 krb5kdc[20539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577216, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:36.950299 20780 init.cc:377] Logged in from keytab as kudu/127.16.157.66@KRBTEST.COM (short username kudu)
I20260928 06:33:36.950911 20780 webserver.cc:460] Webserver started at http://127.16.157.66:39753/ using document root <none> and password file <none>
I20260928 06:33:36.951205 20780 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:36.951275 20780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:36.951421 20780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:36.952894 20780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/minicluster-data/ts-1/data/instance:
uuid: "2fe319ef940046978b7d7824eebfc10d"
format_stamp: "Formatted at 2026-09-28 06:33:36 on dist-test-slave-rrj5"
server_key: "0acd3156e53b442977c6b2bb9d23ba8c"
server_key_iv: "5e8bdde9a1ce860bc3a81ab399f32d11"
server_key_version: "encryptionkey@0"
I20260928 06:33:36.953338 20780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/minicluster-data/ts-1/wal/instance:
uuid: "2fe319ef940046978b7d7824eebfc10d"
format_stamp: "Formatted at 2026-09-28 06:33:36 on dist-test-slave-rrj5"
server_key: "0acd3156e53b442977c6b2bb9d23ba8c"
server_key_iv: "5e8bdde9a1ce860bc3a81ab399f32d11"
server_key_version: "encryptionkey@0"
I20260928 06:33:36.955261 20780 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260928 06:33:36.956362 20795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:36.956554 20780 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:36.956668 20780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/minicluster-data/ts-1/wal
uuid: "2fe319ef940046978b7d7824eebfc10d"
format_stamp: "Formatted at 2026-09-28 06:33:36 on dist-test-slave-rrj5"
server_key: "0acd3156e53b442977c6b2bb9d23ba8c"
server_key_iv: "5e8bdde9a1ce860bc3a81ab399f32d11"
server_key_version: "encryptionkey@0"
I20260928 06:33:36.956748 20780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:36.985229 20780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:36.988085 20780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:36.988229 20780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:36.988471 20780 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:33:36.988811 20780 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:33:36.988852 20780 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:36.988888 20780 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:33:36.988907 20780 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:36.996091 20780 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.66:33679
I20260928 06:33:36.996168 20908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.66:33679 every 8 connection(s)
I20260928 06:33:36.996490 20780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
Sep 28 06:33:37 dist-test-slave-rrj5 krb5kdc[20539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577216, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.66@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:33:37.005676 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 20780
I20260928 06:33:37.005801 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/minicluster-data/ts-1/wal/instance
I20260928 06:33:37.006016 17013 external_mini_cluster.cc:1428] Setting key 20e71b7ccf116e035dec9891b70990a6
I20260928 06:33:37.010821 20623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:36.997371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.66:36625 (local address 127.16.157.126:43035)
0928 06:33:36.997436 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:36.997438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:36.997684 (+ 246us) server_negotiation.cc:407] Connection header received
0928 06:33:36.999266 (+ 1582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:36.999269 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:36.999284 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:36.999319 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:37.000419 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:37.001332 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:37.003055 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:37.003322 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:37.003409 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:37.005916 (+ 2507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:37.005917 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:37.005918 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:37.005926 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:37.009065 (+ 3139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:37.009643 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:37.009645 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:37.009646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:37.009712 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:37.009995 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:37.009996 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:37.009996 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:37.010136 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:37.010189 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:37.010565 (+ 376us) server_negotiation.cc:299] Negotiation successful
0928 06:33:37.010694 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 06:33:37.011382 20911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:36.997338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:43035 (local address 127.16.157.66:36625)
0928 06:33:36.997511 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:36.997565 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:36.999084 (+ 1519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:36.999530 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:36.999536 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:36.999953 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:37.000289 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:37.000297 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:37.001462 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:37.001463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:37.002966 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:37.002969 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:37.003419 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:37.003420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:37.003553 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:37.004106 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:37.004115 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:37.005783 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:37.009169 (+ 3386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:37.009176 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:37.009178 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:37.009509 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:37.009782 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:37.009783 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:37.009783 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:37.009922 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:37.010223 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:37.010229 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:37.010397 (+ 168us) client_negotiation.cc:769] Sending connection context
0928 06:33:37.010933 (+ 536us) client_negotiation.cc:240] Negotiation successful
0928 06:33:37.011134 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":87,"threads_started":1}
I20260928 06:33:37.011945 20909 heartbeater.cc:347] Connected to a master server at 127.16.157.126:43035
I20260928 06:33:37.012054 20909 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:37.012280 20909 heartbeater.cc:511] Master 127.16.157.126:43035 requested a full tablet report, sending...
I20260928 06:33:37.012703 20584 ts_manager.cc:194] Registered new tserver with Master: 2fe319ef940046978b7d7824eebfc10d (127.16.157.66:33679)
I20260928 06:33:37.013216 20584 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.157.66@KRBTEST.COM'} at 127.16.157.66:36625
WARNING: no policy specified for kudu/127.16.157.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab.
Entry for principal kudu/127.16.157.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab.
I20260928 06:33:37.036288 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.67:0
--local_ip_for_outbound_sockets=127.16.157.67
--webserver_interface=127.16.157.67
--webserver_port=0
--tserver_master_addrs=127.16.157.126:43035
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab
--principal=kudu/127.16.157.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:38785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:37.135957 20914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:37.136212 20914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:37.136252 20914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:37.137388 20914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:37.137454 20914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:37.137497 20914 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.67
I20260928 06:33:37.139277 20914 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:38785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab
--principal=kudu/127.16.157.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.157.67
--webserver_port=0
--tserver_master_addrs=127.16.157.126:43035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20914
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.67
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:37.139535 20914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:37.139750 20914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:37.144881 20919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:37.144984 20920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:37.144855 20922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:37.145116 20914 server_base.cc:1034] running on GCE node
I20260928 06:33:37.145310 20914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:37.145612 20914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:37.146781 20914 hybrid_clock.cc:648] HybridClock initialized: now 1790577217146766 us; error 30 us; skew 500 ppm
Sep 28 06:33:37 dist-test-slave-rrj5 krb5kdc[20539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577217, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:37.149828 20914 init.cc:377] Logged in from keytab as kudu/127.16.157.67@KRBTEST.COM (short username kudu)
I20260928 06:33:37.150306 20914 webserver.cc:460] Webserver started at http://127.16.157.67:45429/ using document root <none> and password file <none>
I20260928 06:33:37.150568 20914 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:37.150628 20914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:37.150750 20914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:37.152087 20914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/minicluster-data/ts-2/data/instance:
uuid: "be452138d501482bbdd6e9b2fdb6811e"
format_stamp: "Formatted at 2026-09-28 06:33:37 on dist-test-slave-rrj5"
server_key: "17793c728ff98c5eba612888979b661b"
server_key_iv: "2511810aad5553032ef1866de683e980"
server_key_version: "encryptionkey@0"
I20260928 06:33:37.152484 20914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/minicluster-data/ts-2/wal/instance:
uuid: "be452138d501482bbdd6e9b2fdb6811e"
format_stamp: "Formatted at 2026-09-28 06:33:37 on dist-test-slave-rrj5"
server_key: "17793c728ff98c5eba612888979b661b"
server_key_iv: "2511810aad5553032ef1866de683e980"
server_key_version: "encryptionkey@0"
I20260928 06:33:37.154096 20914 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:33:37.154896 20929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:37.155086 20914 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:33:37.155174 20914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/minicluster-data/ts-2/wal
uuid: "be452138d501482bbdd6e9b2fdb6811e"
format_stamp: "Formatted at 2026-09-28 06:33:37 on dist-test-slave-rrj5"
server_key: "17793c728ff98c5eba612888979b661b"
server_key_iv: "2511810aad5553032ef1866de683e980"
server_key_version: "encryptionkey@0"
I20260928 06:33:37.155243 20914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:37.171213 20914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:37.173756 20914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:37.173892 20914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:37.174114 20914 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:33:37.174408 20914 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:33:37.174459 20914 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:37.174487 20914 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:33:37.174511 20914 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:37.181790 20914 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.67:32933
I20260928 06:33:37.181964 21042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.67:32933 every 8 connection(s)
I20260928 06:33:37.182534 20914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
Sep 28 06:33:37 dist-test-slave-rrj5 krb5kdc[20539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577217, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.67@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:33:37.191967 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 20914
I20260928 06:33:37.192063 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577196946045-17013-0/minicluster-data/ts-2/wal/instance
I20260928 06:33:37.192253 17013 external_mini_cluster.cc:1428] Setting key 3d531658a5d3a674904b02a2bdb14c31
I20260928 06:33:37.193871 20623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:37.183722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:52181 (local address 127.16.157.126:43035)
0928 06:33:37.183764 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:37.183766 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:37.183778 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:33:37.184904 (+ 1126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:37.184905 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:37.184937 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:37.185015 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:37.185794 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:37.186565 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:37.187555 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:37.187775 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:37.187852 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:37.190157 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:37.190158 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:37.190159 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:37.190166 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:37.191732 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:37.192800 (+ 1068us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:37.192801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:37.192801 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:37.192869 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:37.193145 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:37.193147 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:37.193147 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:37.193292 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:37.193363 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:37.193695 (+ 332us) server_negotiation.cc:299] Negotiation successful
0928 06:33:37.193777 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:33:37.194049 21045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:37.183405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:43035 (local address 127.16.157.67:52181)
0928 06:33:37.183608 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:37.183653 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:37.184792 (+ 1139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:37.185067 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:37.185071 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:37.185394 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:37.185681 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:37.185687 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:37.186663 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:37.186664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:37.187462 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:37.187465 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:37.187859 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:37.187860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:37.187984 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:37.188461 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:37.188469 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:37.190035 (+ 1566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:37.192407 (+ 2372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:37.192413 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:37.192416 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:37.192703 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:37.192958 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:37.192959 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:37.192959 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:37.193088 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:37.193363 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:37.193374 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:37.193529 (+ 155us) client_negotiation.cc:769] Sending connection context
0928 06:33:37.193661 (+ 132us) client_negotiation.cc:240] Negotiation successful
0928 06:33:37.193817 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":92,"threads_started":1}
I20260928 06:33:37.194456 21043 heartbeater.cc:347] Connected to a master server at 127.16.157.126:43035
I20260928 06:33:37.194566 21043 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:37.194758 21043 heartbeater.cc:511] Master 127.16.157.126:43035 requested a full tablet report, sending...
I20260928 06:33:37.195081 20584 ts_manager.cc:194] Registered new tserver with Master: be452138d501482bbdd6e9b2fdb6811e (127.16.157.67:32933)
I20260928 06:33:37.195534 20584 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.157.67@KRBTEST.COM'} at 127.16.157.67:52181
I20260928 06:33:37.203806 17013 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 06:33:37 dist-test-slave-rrj5 krb5kdc[20539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 06:33:37 dist-test-slave-rrj5 krb5kdc[20539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.157.65@KRBTEST.COM
I20260928 06:33:37.234999 21050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:37.224794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50070 (local address 127.16.157.65:34085)
0928 06:33:37.225000 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:37.225006 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:37.225016 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:33:37.225044 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:37.225046 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:37.225063 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:37.225135 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:37.225956 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:37.227156 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:37.228192 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:37.228672 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:37.228742 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:37.231689 (+ 2947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:37.231699 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:37.231699 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:37.231706 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:37.233634 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:37.234058 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:37.234062 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:37.234062 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:37.234164 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:37.234363 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:37.234364 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:37.234364 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:37.234549 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:37.234703 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:37.234774 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 06:33:37.234860 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":114,"threads_started":1}
Sep 28 06:33:37 dist-test-slave-rrj5 krb5kdc[20539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:33:37.244794 20623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:37.236343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35712 (local address 127.16.157.126:43035)
0928 06:33:37.236394 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:37.236396 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:37.236408 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:33:37.236488 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:37.236489 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:37.236499 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:37.236555 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:37.237160 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:37.237955 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:37.238971 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:37.239294 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:37.239377 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:37.241987 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:37.241988 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:37.241989 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:37.241997 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:37.243596 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:37.243980 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:37.243982 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:37.243982 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:37.244044 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:37.244252 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:37.244253 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:37.244253 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:37.244407 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:37.244515 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:37.244617 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 06:33:37.244694 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 06:33:37.245023 20584 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35712
I20260928 06:33:37.245435 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 20646
I20260928 06:33:37.250545 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 20780
I20260928 06:33:37.257380 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 20914
I20260928 06:33:37.265877 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 20552
2026-09-28T06:33:37Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2117 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:33:37 dist-test-slave-rrj5 krb5kdc[21055](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:33:37 dist-test-slave-rrj5 krb5kdc[21055](info): set up 2 sockets
Sep 28 06:33:37 dist-test-slave-rrj5 krb5kdc[21055](info): commencing operation
krb5kdc: starting...
W20260928 06:33:38.382331 17013 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Sep 28 06:33:38 dist-test-slave-rrj5 krb5kdc[21055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:33:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:33:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.157.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab.
Entry for principal kudu/127.16.157.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab.
I20260928 06:33:38.504009 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:39751
--webserver_interface=127.16.157.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=kudu/127.16.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:43377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.157.126:39751
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:38.583696 21068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:38.583926 21068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:38.583963 21068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:38.585107 21068 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:33:38.585178 21068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:38.585207 21068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:38.585238 21068 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:33:38.585263 21068 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:33:38.586771 21068 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:43377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.157.126:39751
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=kudu/127.16.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:39751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21068
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:38.587046 21068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:38.587296 21068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:38.592635 21074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:38.592641 21073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:38.592826 21068 server_base.cc:1034] running on GCE node
W20260928 06:33:38.592644 21076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:38.593178 21068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:38.593497 21068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:38.594662 21068 hybrid_clock.cc:648] HybridClock initialized: now 1790577218594657 us; error 37 us; skew 500 ppm
Sep 28 06:33:38 dist-test-slave-rrj5 krb5kdc[21055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577218, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:38.597424 21068 init.cc:377] Logged in from keytab as kudu/127.16.157.126@KRBTEST.COM (short username kudu)
I20260928 06:33:38.597820 21068 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:38.597869 21068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:38.597999 21068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:38.599028 21068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/minicluster-data/master-0/data/instance:
uuid: "79bf49e53bb24800b20a6984d9124269"
format_stamp: "Formatted at 2026-09-28 06:33:38 on dist-test-slave-rrj5"
server_key: "2c3071ec476a0c3eb352edca30f01b75"
server_key_iv: "44fdcc18d2dde002d15749858ea6a9d7"
server_key_version: "encryptionkey@0"
I20260928 06:33:38.599390 21068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/minicluster-data/master-0/wal/instance:
uuid: "79bf49e53bb24800b20a6984d9124269"
format_stamp: "Formatted at 2026-09-28 06:33:38 on dist-test-slave-rrj5"
server_key: "2c3071ec476a0c3eb352edca30f01b75"
server_key_iv: "44fdcc18d2dde002d15749858ea6a9d7"
server_key_version: "encryptionkey@0"
I20260928 06:33:38.600950 21068 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:33:38.601786 21082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:38.601966 21068 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:33:38.602068 21068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/minicluster-data/master-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/minicluster-data/master-0/wal
uuid: "79bf49e53bb24800b20a6984d9124269"
format_stamp: "Formatted at 2026-09-28 06:33:38 on dist-test-slave-rrj5"
server_key: "2c3071ec476a0c3eb352edca30f01b75"
server_key_iv: "44fdcc18d2dde002d15749858ea6a9d7"
server_key_version: "encryptionkey@0"
I20260928 06:33:38.602178 21068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:38.620076 21068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:38.622350 21068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:38.622519 21068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:38.626830 21068 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.126:39751
I20260928 06:33:38.626847 21134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.126:39751 every 8 connection(s)
I20260928 06:33:38.627174 21068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
I20260928 06:33:38.628244 21135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:38.628638 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 21068
I20260928 06:33:38.628746 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/minicluster-data/master-0/wal/instance
I20260928 06:33:38.628935 17013 external_mini_cluster.cc:1428] Setting key 061a5bc66d4026149978c7e01ada315f
I20260928 06:33:38.631302 21135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79bf49e53bb24800b20a6984d9124269: Bootstrap starting.
I20260928 06:33:38.631991 21135 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79bf49e53bb24800b20a6984d9124269: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:38.632287 21135 log.cc:824] T 00000000000000000000000000000000 P 79bf49e53bb24800b20a6984d9124269: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:38.633025 21135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79bf49e53bb24800b20a6984d9124269: No bootstrap required, opened a new log
Sep 28 06:33:38 dist-test-slave-rrj5 krb5kdc[21055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:33:38.635293 21135 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79bf49e53bb24800b20a6984d9124269 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79bf49e53bb24800b20a6984d9124269" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 39751 } }
I20260928 06:33:38.635447 21135 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79bf49e53bb24800b20a6984d9124269 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:38.635481 21135 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79bf49e53bb24800b20a6984d9124269 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79bf49e53bb24800b20a6984d9124269, State: Initialized, Role: FOLLOWER
I20260928 06:33:38.635635 21135 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79bf49e53bb24800b20a6984d9124269 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79bf49e53bb24800b20a6984d9124269" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 39751 } }
I20260928 06:33:38.635708 21135 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79bf49e53bb24800b20a6984d9124269 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:33:38.635737 21135 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79bf49e53bb24800b20a6984d9124269 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:33:38.635779 21135 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79bf49e53bb24800b20a6984d9124269 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:38.636402 21135 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79bf49e53bb24800b20a6984d9124269 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79bf49e53bb24800b20a6984d9124269" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 39751 } }
I20260928 06:33:38.636505 21135 leader_election.cc:302] T 00000000000000000000000000000000 P 79bf49e53bb24800b20a6984d9124269 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79bf49e53bb24800b20a6984d9124269; no voters:
I20260928 06:33:38.636650 21135 leader_election.cc:288] T 00000000000000000000000000000000 P 79bf49e53bb24800b20a6984d9124269 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:33:38.636888 21135 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79bf49e53bb24800b20a6984d9124269 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:33:38.636751 21140 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79bf49e53bb24800b20a6984d9124269 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:33:38.637205 21140 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79bf49e53bb24800b20a6984d9124269 [term 1 LEADER]: Becoming Leader. State: Replica: 79bf49e53bb24800b20a6984d9124269, State: Running, Role: LEADER
I20260928 06:33:38.637285 21140 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79bf49e53bb24800b20a6984d9124269 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79bf49e53bb24800b20a6984d9124269" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 39751 } }
I20260928 06:33:38.637645 21141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79bf49e53bb24800b20a6984d9124269 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79bf49e53bb24800b20a6984d9124269" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79bf49e53bb24800b20a6984d9124269" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 39751 } } }
I20260928 06:33:38.637734 21141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79bf49e53bb24800b20a6984d9124269 [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:38.637717 21142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79bf49e53bb24800b20a6984d9124269 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79bf49e53bb24800b20a6984d9124269. Latest consensus state: current_term: 1 leader_uuid: "79bf49e53bb24800b20a6984d9124269" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79bf49e53bb24800b20a6984d9124269" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 39751 } } }
I20260928 06:33:38.637807 21142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79bf49e53bb24800b20a6984d9124269 [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:38.637962 21151 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:33:38.638470 21151 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:33:38.640143 21138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:38.629436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53754 (local address 127.16.157.126:39751)
0928 06:33:38.629649 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:38.629657 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:38.629704 (+ 47us) server_negotiation.cc:407] Connection header received
0928 06:33:38.630107 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:38.630141 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:38.630185 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:38.630340 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:38.630949 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:38.631943 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:38.632986 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:38.633461 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:38.633600 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:38.635932 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:38.635935 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:38.635936 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:38.635947 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:38.638091 (+ 2144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:38.638622 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:38.638624 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:38.638625 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:38.638722 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:38.639003 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:38.639004 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:38.639004 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:38.639304 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:38.639476 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:38.639794 (+ 318us) server_negotiation.cc:299] Negotiation successful
0928 06:33:38.639933 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":101,"threads_started":1}
I20260928 06:33:38.640996 21151 catalog_manager.cc:1347] Generated new cluster ID: 29658aa2a86a41988b0e2ad3aa0c47f8
I20260928 06:33:38.641047 21151 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:33:38.650036 21151 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:33:38.650574 21151 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:33:38.660959 21151 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79bf49e53bb24800b20a6984d9124269: Generated new TSK 0
I20260928 06:33:38.661152 21151 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.157.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab.
Entry for principal kudu/127.16.157.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab.
I20260928 06:33:38.690706 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.65:0
--local_ip_for_outbound_sockets=127.16.157.65
--webserver_interface=127.16.157.65
--webserver_port=0
--tserver_master_addrs=127.16.157.126:39751
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab
--principal=kudu/127.16.157.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:43377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:38.771027 21161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:38.771273 21161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:38.771315 21161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:38.772478 21161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:38.772547 21161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:38.772595 21161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.65
I20260928 06:33:38.774183 21161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:43377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab
--principal=kudu/127.16.157.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.157.65
--webserver_port=0
--tserver_master_addrs=127.16.157.126:39751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.65
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:38.774463 21161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:38.774708 21161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:38.779731 21169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:38.779719 21167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:38.780153 21166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:38.780673 21161 server_base.cc:1034] running on GCE node
I20260928 06:33:38.780900 21161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:38.781172 21161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:38.782328 21161 hybrid_clock.cc:648] HybridClock initialized: now 1790577218782317 us; error 30 us; skew 500 ppm
Sep 28 06:33:38 dist-test-slave-rrj5 krb5kdc[21055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577218, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:38.784925 21161 init.cc:377] Logged in from keytab as kudu/127.16.157.65@KRBTEST.COM (short username kudu)
I20260928 06:33:38.785305 21161 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:38.785378 21161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:38.785521 21161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:38.786768 21161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/minicluster-data/ts-0/data/instance:
uuid: "35b5a1fc2db54cde806ce230c0f1e472"
format_stamp: "Formatted at 2026-09-28 06:33:38 on dist-test-slave-rrj5"
server_key: "f146a2ccc48566e491921e7033214120"
server_key_iv: "036b44890ad427b3a72485d74e7e6567"
server_key_version: "encryptionkey@0"
I20260928 06:33:38.787175 21161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/minicluster-data/ts-0/wal/instance:
uuid: "35b5a1fc2db54cde806ce230c0f1e472"
format_stamp: "Formatted at 2026-09-28 06:33:38 on dist-test-slave-rrj5"
server_key: "f146a2ccc48566e491921e7033214120"
server_key_iv: "036b44890ad427b3a72485d74e7e6567"
server_key_version: "encryptionkey@0"
I20260928 06:33:38.788617 21161 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:33:38.789381 21175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:38.789532 21161 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:33:38.789649 21161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/minicluster-data/ts-0/wal
uuid: "35b5a1fc2db54cde806ce230c0f1e472"
format_stamp: "Formatted at 2026-09-28 06:33:38 on dist-test-slave-rrj5"
server_key: "f146a2ccc48566e491921e7033214120"
server_key_iv: "036b44890ad427b3a72485d74e7e6567"
server_key_version: "encryptionkey@0"
I20260928 06:33:38.789754 21161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:38.805917 21161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:38.808230 21161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:38.808391 21161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:38.808616 21161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:33:38.808931 21161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:33:38.808991 21161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:38.809036 21161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:33:38.809063 21161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:38.815487 21161 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.65:36493
I20260928 06:33:38.815562 21288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.65:36493 every 8 connection(s)
I20260928 06:33:38.815846 21161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
Sep 28 06:33:38 dist-test-slave-rrj5 krb5kdc[21055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577218, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.65@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:33:38.825241 21138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:38.816580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:53873 (local address 127.16.157.126:39751)
0928 06:33:38.816628 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:38.816630 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:38.816842 (+ 212us) server_negotiation.cc:407] Connection header received
0928 06:33:38.817375 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:38.817377 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:38.817389 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:38.817443 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:38.818468 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:38.818965 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:38.819955 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:38.820177 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:38.820253 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:38.822537 (+ 2284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:38.822538 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:38.822538 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:38.822544 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:38.823869 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:38.824329 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:38.824329 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:38.824330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:38.824379 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:38.824632 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:38.824633 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:38.824633 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:38.824731 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:38.824780 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:38.825108 (+ 328us) server_negotiation.cc:299] Negotiation successful
0928 06:33:38.825176 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 06:33:38.825450 21291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:38.816554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:39751 (local address 127.16.157.65:53873)
0928 06:33:38.816724 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:38.816759 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:38.817251 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:38.817520 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:38.817523 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:38.817847 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:38.818352 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:38.818357 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:38.819067 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:38.819068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:38.819863 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:38.819866 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:38.820260 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:38.820261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:38.820384 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:38.820921 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:38.820928 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:38.822441 (+ 1513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:38.823952 (+ 1511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:38.823959 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:38.823961 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:38.824253 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:38.824443 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:38.824444 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:38.824444 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:38.824570 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:38.824793 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:38.824797 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:38.824956 (+ 159us) client_negotiation.cc:769] Sending connection context
0928 06:33:38.825089 (+ 133us) client_negotiation.cc:240] Negotiation successful
0928 06:33:38.825244 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":96,"threads_started":1}
I20260928 06:33:38.825619 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 21161
I20260928 06:33:38.825757 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/minicluster-data/ts-0/wal/instance
I20260928 06:33:38.825937 17013 external_mini_cluster.cc:1428] Setting key db6c88e6eeaf4ccebbb8345a190b6b0a
I20260928 06:33:38.826191 21289 heartbeater.cc:347] Connected to a master server at 127.16.157.126:39751
I20260928 06:33:38.826287 21289 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:38.826428 21289 heartbeater.cc:511] Master 127.16.157.126:39751 requested a full tablet report, sending...
I20260928 06:33:38.828109 21099 ts_manager.cc:194] Registered new tserver with Master: 35b5a1fc2db54cde806ce230c0f1e472 (127.16.157.65:36493)
I20260928 06:33:38.829012 21099 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.157.65@KRBTEST.COM'} at 127.16.157.65:53873
WARNING: no policy specified for kudu/127.16.157.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab.
Entry for principal kudu/127.16.157.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab.
I20260928 06:33:38.852178 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.66:0
--local_ip_for_outbound_sockets=127.16.157.66
--webserver_interface=127.16.157.66
--webserver_port=0
--tserver_master_addrs=127.16.157.126:39751
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab
--principal=kudu/127.16.157.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:43377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:38.940577 21294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:38.940816 21294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:38.940855 21294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:38.942068 21294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:38.942131 21294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:38.942178 21294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.66
I20260928 06:33:38.943740 21294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:43377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab
--principal=kudu/127.16.157.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.157.66
--webserver_port=0
--tserver_master_addrs=127.16.157.126:39751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.66
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:38.944011 21294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:38.944249 21294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:38.949337 21299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:38.949347 21300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:38.949348 21302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:38.949623 21294 server_base.cc:1034] running on GCE node
I20260928 06:33:38.949818 21294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:38.950088 21294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:38.951257 21294 hybrid_clock.cc:648] HybridClock initialized: now 1790577218951232 us; error 40 us; skew 500 ppm
Sep 28 06:33:38 dist-test-slave-rrj5 krb5kdc[21055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577218, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:38.954630 21294 init.cc:377] Logged in from keytab as kudu/127.16.157.66@KRBTEST.COM (short username kudu)
I20260928 06:33:38.955009 21294 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:38.955060 21294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:38.955201 21294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:38.956276 21294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/minicluster-data/ts-1/data/instance:
uuid: "d0163b4acb154ae284884b8295034833"
format_stamp: "Formatted at 2026-09-28 06:33:38 on dist-test-slave-rrj5"
server_key: "eebb347f7821ce96213c65583e70a914"
server_key_iv: "7b2a4e8c6e17f89681c58a39a062fd88"
server_key_version: "encryptionkey@0"
I20260928 06:33:38.956636 21294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/minicluster-data/ts-1/wal/instance:
uuid: "d0163b4acb154ae284884b8295034833"
format_stamp: "Formatted at 2026-09-28 06:33:38 on dist-test-slave-rrj5"
server_key: "eebb347f7821ce96213c65583e70a914"
server_key_iv: "7b2a4e8c6e17f89681c58a39a062fd88"
server_key_version: "encryptionkey@0"
I20260928 06:33:38.958070 21294 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 06:33:38.958870 21308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:38.959048 21294 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:38.959156 21294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/minicluster-data/ts-1/wal
uuid: "d0163b4acb154ae284884b8295034833"
format_stamp: "Formatted at 2026-09-28 06:33:38 on dist-test-slave-rrj5"
server_key: "eebb347f7821ce96213c65583e70a914"
server_key_iv: "7b2a4e8c6e17f89681c58a39a062fd88"
server_key_version: "encryptionkey@0"
I20260928 06:33:38.959237 21294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:38.973836 21294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:38.975888 21294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:38.976063 21294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:38.976301 21294 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:33:38.976604 21294 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:33:38.976658 21294 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:38.976699 21294 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:33:38.976727 21294 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:38.983297 21294 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.66:40147
I20260928 06:33:38.983390 21421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.66:40147 every 8 connection(s)
I20260928 06:33:38.983623 21294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
I20260928 06:33:38.987120 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 21294
I20260928 06:33:38.987237 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/minicluster-data/ts-1/wal/instance
I20260928 06:33:38.987430 17013 external_mini_cluster.cc:1428] Setting key c4911e55520be4bc0b164f72145a833e
Sep 28 06:33:38 dist-test-slave-rrj5 krb5kdc[21055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577218, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.66@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:33:38.994237 21138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:38.984361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.66:59495 (local address 127.16.157.126:39751)
0928 06:33:38.984406 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:38.984409 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:38.984639 (+ 230us) server_negotiation.cc:407] Connection header received
0928 06:33:38.985157 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:38.985159 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:38.985171 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:38.985226 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:38.986230 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:38.986737 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:38.988503 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:38.988782 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:38.988868 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:38.991178 (+ 2310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:38.991180 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:38.991180 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:38.991188 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:38.992717 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:38.993155 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:38.993156 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:38.993157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:38.993222 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:38.993523 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:38.993524 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:38.993525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:38.993697 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:38.993773 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:38.994056 (+ 283us) server_negotiation.cc:299] Negotiation successful
0928 06:33:38.994144 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:33:38.994437 21424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:38.984334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:39751 (local address 127.16.157.66:59495)
0928 06:33:38.984510 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:38.984545 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:38.985044 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:38.985303 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:38.985306 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:38.985554 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:38.986123 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:38.986128 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:38.986849 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:38.986850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:38.988406 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:38.988409 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:38.988883 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:38.988884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:38.989040 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:38.989648 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:38.989659 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:38.991069 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:38.992819 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:38.992826 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:38.992829 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:38.993068 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:38.993289 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:38.993290 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:38.993290 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:38.993406 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:38.993775 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:38.993781 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:38.993922 (+ 141us) client_negotiation.cc:769] Sending connection context
0928 06:33:38.994033 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 06:33:38.994203 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":103,"threads_started":1}
I20260928 06:33:38.994935 21422 heartbeater.cc:347] Connected to a master server at 127.16.157.126:39751
I20260928 06:33:38.995041 21422 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:38.995196 21422 heartbeater.cc:511] Master 127.16.157.126:39751 requested a full tablet report, sending...
I20260928 06:33:38.995568 21099 ts_manager.cc:194] Registered new tserver with Master: d0163b4acb154ae284884b8295034833 (127.16.157.66:40147)
I20260928 06:33:38.995941 21099 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.157.66@KRBTEST.COM'} at 127.16.157.66:59495
WARNING: no policy specified for kudu/127.16.157.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab.
Entry for principal kudu/127.16.157.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab.
I20260928 06:33:39.016213 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.67:0
--local_ip_for_outbound_sockets=127.16.157.67
--webserver_interface=127.16.157.67
--webserver_port=0
--tserver_master_addrs=127.16.157.126:39751
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab
--principal=kudu/127.16.157.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:43377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:39.114282 21427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:39.114480 21427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:39.114503 21427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:39.115672 21427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:39.115715 21427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:39.115741 21427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.67
I20260928 06:33:39.117244 21427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:43377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab
--principal=kudu/127.16.157.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.157.67
--webserver_port=0
--tserver_master_addrs=127.16.157.126:39751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.67
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:39.117489 21427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:39.117744 21427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:33:39.122792 21427 server_base.cc:1034] running on GCE node
W20260928 06:33:39.122695 21432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:39.122840 21435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:39.122694 21433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:39.123231 21427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:39.123534 21427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:39.124696 21427 hybrid_clock.cc:648] HybridClock initialized: now 1790577219124675 us; error 37 us; skew 500 ppm
Sep 28 06:33:39 dist-test-slave-rrj5 krb5kdc[21055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577219, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:39.127377 21427 init.cc:377] Logged in from keytab as kudu/127.16.157.67@KRBTEST.COM (short username kudu)
I20260928 06:33:39.127766 21427 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:39.127851 21427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:39.128000 21427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:39.129065 21427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/minicluster-data/ts-2/data/instance:
uuid: "e003c03f280a41d88a53173792bc21bf"
format_stamp: "Formatted at 2026-09-28 06:33:39 on dist-test-slave-rrj5"
server_key: "a3307000898c607a2033da09cef66c81"
server_key_iv: "8f981fbf7e8c1d9698652112d798b6a1"
server_key_version: "encryptionkey@0"
I20260928 06:33:39.129434 21427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/minicluster-data/ts-2/wal/instance:
uuid: "e003c03f280a41d88a53173792bc21bf"
format_stamp: "Formatted at 2026-09-28 06:33:39 on dist-test-slave-rrj5"
server_key: "a3307000898c607a2033da09cef66c81"
server_key_iv: "8f981fbf7e8c1d9698652112d798b6a1"
server_key_version: "encryptionkey@0"
I20260928 06:33:39.130865 21427 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:33:39.131676 21441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:39.131861 21427 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:33:39.131953 21427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/minicluster-data/ts-2/wal
uuid: "e003c03f280a41d88a53173792bc21bf"
format_stamp: "Formatted at 2026-09-28 06:33:39 on dist-test-slave-rrj5"
server_key: "a3307000898c607a2033da09cef66c81"
server_key_iv: "8f981fbf7e8c1d9698652112d798b6a1"
server_key_version: "encryptionkey@0"
I20260928 06:33:39.132040 21427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:39.143213 21427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:39.145397 21427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:39.145570 21427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:39.145846 21427 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:33:39.146163 21427 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:33:39.146219 21427 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:39.146296 21427 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:33:39.146335 21427 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:39.153309 21427 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.67:44809
I20260928 06:33:39.153337 21554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.67:44809 every 8 connection(s)
I20260928 06:33:39.153697 21427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
Sep 28 06:33:39 dist-test-slave-rrj5 krb5kdc[21055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577219, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.67@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:33:39.161736 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 21427
I20260928 06:33:39.161876 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/minicluster-data/ts-2/wal/instance
I20260928 06:33:39.162047 17013 external_mini_cluster.cc:1428] Setting key 891a5a2aa3a64a500a19f023e4dc46ab
I20260928 06:33:39.164584 21138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:39.154429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:50249 (local address 127.16.157.126:39751)
0928 06:33:39.154510 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:39.154513 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:39.154750 (+ 237us) server_negotiation.cc:407] Connection header received
0928 06:33:39.155444 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:39.155445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:39.155454 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:39.155496 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:39.156820 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:39.157320 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:39.158694 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:39.158882 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:39.158938 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:39.161297 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:39.161298 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:39.161299 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:39.161306 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:39.163104 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:39.163618 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:39.163620 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:39.163620 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:39.163685 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:39.163938 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:39.163939 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:39.163939 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:39.164057 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:39.164135 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:39.164415 (+ 280us) server_negotiation.cc:299] Negotiation successful
0928 06:33:39.164497 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 06:33:39.164765 21557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:39.154394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:39751 (local address 127.16.157.67:50249)
0928 06:33:39.154582 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:39.154641 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:39.155333 (+ 692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:39.155598 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:39.155603 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:39.155988 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:39.156716 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:39.156723 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:39.157424 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:39.157426 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:39.158611 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:39.158614 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:39.158976 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:39.158977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:39.159130 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:39.159642 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:39.159652 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:39.161202 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:39.163196 (+ 1994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:39.163203 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:39.163205 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:39.163531 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:39.163751 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:39.163752 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:39.163753 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:39.163870 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:39.164127 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:39.164133 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:39.164276 (+ 143us) client_negotiation.cc:769] Sending connection context
0928 06:33:39.164379 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 06:33:39.164535 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":103,"threads_started":1}
I20260928 06:33:39.165378 21555 heartbeater.cc:347] Connected to a master server at 127.16.157.126:39751
I20260928 06:33:39.165468 21555 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:39.165661 21555 heartbeater.cc:511] Master 127.16.157.126:39751 requested a full tablet report, sending...
I20260928 06:33:39.166039 21099 ts_manager.cc:194] Registered new tserver with Master: e003c03f280a41d88a53173792bc21bf (127.16.157.67:44809)
I20260928 06:33:39.166363 21099 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.157.67@KRBTEST.COM'} at 127.16.157.67:50249
I20260928 06:33:39.174511 17013 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:33:39.183059 21138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:39.176027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53760 (local address 127.16.157.126:39751)
0928 06:33:39.176093 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:39.176095 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:39.176202 (+ 107us) server_negotiation.cc:407] Connection header received
0928 06:33:39.176268 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:39.176269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:39.176279 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:39.176341 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:39.177046 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:39.177514 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:39.178614 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:39.178792 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:39.178838 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:39.180307 (+ 1469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:39.180309 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:39.180309 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:39.180316 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:39.181828 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:39.182212 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:39.182213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:39.182214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:39.182275 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:39.182490 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:39.182491 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:39.182491 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:39.182641 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:39.182747 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:39.182881 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 06:33:39.182956 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 06:33:39.184634 21099 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53760:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:33:39.186260 21099 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:33:39.193614 21566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:39.188717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.65:36493 (local address 127.0.0.1:59846)
0928 06:33:39.188948 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:39.188991 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:39.189062 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:39.189541 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:39.189548 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:39.189551 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:33:39.189670 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:39.189672 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:39.190980 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:39.190981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:39.192318 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:39.192321 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:39.193260 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:39.193261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:39.193346 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:39.193361 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 06:33:39.193408 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 06:33:39.193506 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":4,"thread_start_us":66,"threads_started":1}
I20260928 06:33:39.193833 21568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:39.188717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.67:44809 (local address 127.0.0.1:36098)
0928 06:33:39.189187 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:39.189196 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:39.189244 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:39.189559 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:39.189560 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:39.189562 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:33:39.189655 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:39.189657 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:39.190964 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:39.190965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:39.192366 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:39.192369 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:39.193428 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:39.193429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:39.193489 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:39.193492 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:33:39.193539 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 06:33:39.193596 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"mutex_wait_us":4,"thread_start_us":82,"threads_started":1}
I20260928 06:33:39.194077 21567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:39.188735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.66:40147 (local address 127.0.0.1:46260)
0928 06:33:39.189655 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:39.189665 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:39.189717 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:39.189991 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:39.189991 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:39.189994 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:33:39.190027 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:39.190028 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:39.191477 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:39.191479 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:39.192555 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:39.192557 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:39.193854 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:39.193855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:39.193921 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:39.193925 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:33:39.193974 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 06:33:39.194015 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":898,"mutex_wait_us":2,"thread_start_us":92,"threads_started":1}
I20260928 06:33:39.194208 21571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:39.189191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59846 (local address 127.16.157.65:36493)
0928 06:33:39.189351 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:39.189355 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:39.189364 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:33:39.189400 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:39.189402 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:39.189416 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:39.189511 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:33:39.189760 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:39.190858 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:39.192445 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:39.193161 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:39.194008 (+ 847us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:39.194069 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:39.194094 (+ 25us) server_negotiation.cc:299] Negotiation successful
0928 06:33:39.194136 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":81,"threads_started":1}
I20260928 06:33:39.194303 21570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:39.189002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46260 (local address 127.16.157.66:40147)
0928 06:33:39.189411 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:39.189416 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:39.189780 (+ 364us) server_negotiation.cc:407] Connection header received
0928 06:33:39.189824 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:39.189827 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:39.189842 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:39.189911 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:33:39.190102 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:39.191128 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:39.192637 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:39.193359 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:39.193769 (+ 410us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:39.193828 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:39.194172 (+ 344us) server_negotiation.cc:299] Negotiation successful
0928 06:33:39.194213 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":90,"threads_started":1}
I20260928 06:33:39.194833 21223 tablet_service.cc:1467] Processing CreateTablet for tablet 227a2f18a3a745a2904b3b741d854554 (DEFAULT_TABLE table=test-table [id=247d84bcfa5c4d2b99e004570c4146bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:33:39.194959 21356 tablet_service.cc:1467] Processing CreateTablet for tablet 227a2f18a3a745a2904b3b741d854554 (DEFAULT_TABLE table=test-table [id=247d84bcfa5c4d2b99e004570c4146bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:33:39.195179 21223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 227a2f18a3a745a2904b3b741d854554. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:39.195291 21356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 227a2f18a3a745a2904b3b741d854554. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:39.197876 21572 tablet_bootstrap.cc:492] T 227a2f18a3a745a2904b3b741d854554 P 35b5a1fc2db54cde806ce230c0f1e472: Bootstrap starting.
I20260928 06:33:39.198525 21572 tablet_bootstrap.cc:654] T 227a2f18a3a745a2904b3b741d854554 P 35b5a1fc2db54cde806ce230c0f1e472: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:39.198542 21573 tablet_bootstrap.cc:492] T 227a2f18a3a745a2904b3b741d854554 P d0163b4acb154ae284884b8295034833: Bootstrap starting.
I20260928 06:33:39.194096 21569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:39.188977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36098 (local address 127.16.157.67:44809)
0928 06:33:39.189137 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:39.189142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:39.189307 (+ 165us) server_negotiation.cc:407] Connection header received
0928 06:33:39.189355 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:39.189358 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:39.189376 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:39.189449 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:33:39.189743 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:39.190852 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:39.192461 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:39.193276 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:39.193342 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:39.193394 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:39.193652 (+ 258us) server_negotiation.cc:299] Negotiation successful
0928 06:33:39.194016 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":101,"threads_started":1}
I20260928 06:33:39.199232 21573 tablet_bootstrap.cc:654] T 227a2f18a3a745a2904b3b741d854554 P d0163b4acb154ae284884b8295034833: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:39.199275 21572 log.cc:824] T 227a2f18a3a745a2904b3b741d854554 P 35b5a1fc2db54cde806ce230c0f1e472: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:39.199378 21489 tablet_service.cc:1467] Processing CreateTablet for tablet 227a2f18a3a745a2904b3b741d854554 (DEFAULT_TABLE table=test-table [id=247d84bcfa5c4d2b99e004570c4146bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:33:39.199687 21489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 227a2f18a3a745a2904b3b741d854554. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:39.199849 21573 log.cc:824] T 227a2f18a3a745a2904b3b741d854554 P d0163b4acb154ae284884b8295034833: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:39.200166 21572 tablet_bootstrap.cc:492] T 227a2f18a3a745a2904b3b741d854554 P 35b5a1fc2db54cde806ce230c0f1e472: No bootstrap required, opened a new log
I20260928 06:33:39.200248 21572 ts_tablet_manager.cc:1397] T 227a2f18a3a745a2904b3b741d854554 P 35b5a1fc2db54cde806ce230c0f1e472: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 06:33:39.200582 21573 tablet_bootstrap.cc:492] T 227a2f18a3a745a2904b3b741d854554 P d0163b4acb154ae284884b8295034833: No bootstrap required, opened a new log
I20260928 06:33:39.200667 21573 ts_tablet_manager.cc:1397] T 227a2f18a3a745a2904b3b741d854554 P d0163b4acb154ae284884b8295034833: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:33:39.202297 21572 raft_consensus.cc:348] T 227a2f18a3a745a2904b3b741d854554 P 35b5a1fc2db54cde806ce230c0f1e472 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35b5a1fc2db54cde806ce230c0f1e472" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 36493 } } peers { permanent_uuid: "d0163b4acb154ae284884b8295034833" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 40147 } } peers { permanent_uuid: "e003c03f280a41d88a53173792bc21bf" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 44809 } }
I20260928 06:33:39.202466 21572 raft_consensus.cc:374] T 227a2f18a3a745a2904b3b741d854554 P 35b5a1fc2db54cde806ce230c0f1e472 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:39.202517 21572 raft_consensus.cc:729] T 227a2f18a3a745a2904b3b741d854554 P 35b5a1fc2db54cde806ce230c0f1e472 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35b5a1fc2db54cde806ce230c0f1e472, State: Initialized, Role: FOLLOWER
I20260928 06:33:39.202652 21572 consensus_queue.cc:260] T 227a2f18a3a745a2904b3b741d854554 P 35b5a1fc2db54cde806ce230c0f1e472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35b5a1fc2db54cde806ce230c0f1e472" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 36493 } } peers { permanent_uuid: "d0163b4acb154ae284884b8295034833" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 40147 } } peers { permanent_uuid: "e003c03f280a41d88a53173792bc21bf" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 44809 } }
I20260928 06:33:39.202957 21572 ts_tablet_manager.cc:1428] T 227a2f18a3a745a2904b3b741d854554 P 35b5a1fc2db54cde806ce230c0f1e472: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:33:39.203079 21289 heartbeater.cc:503] Master 127.16.157.126:39751 was elected leader, sending a full tablet report...
I20260928 06:33:39.203065 21573 raft_consensus.cc:348] T 227a2f18a3a745a2904b3b741d854554 P d0163b4acb154ae284884b8295034833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35b5a1fc2db54cde806ce230c0f1e472" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 36493 } } peers { permanent_uuid: "d0163b4acb154ae284884b8295034833" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 40147 } } peers { permanent_uuid: "e003c03f280a41d88a53173792bc21bf" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 44809 } }
I20260928 06:33:39.203271 21573 raft_consensus.cc:374] T 227a2f18a3a745a2904b3b741d854554 P d0163b4acb154ae284884b8295034833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:39.203325 21573 raft_consensus.cc:729] T 227a2f18a3a745a2904b3b741d854554 P d0163b4acb154ae284884b8295034833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0163b4acb154ae284884b8295034833, State: Initialized, Role: FOLLOWER
I20260928 06:33:39.203447 21573 consensus_queue.cc:260] T 227a2f18a3a745a2904b3b741d854554 P d0163b4acb154ae284884b8295034833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35b5a1fc2db54cde806ce230c0f1e472" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 36493 } } peers { permanent_uuid: "d0163b4acb154ae284884b8295034833" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 40147 } } peers { permanent_uuid: "e003c03f280a41d88a53173792bc21bf" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 44809 } }
I20260928 06:33:39.203228 21576 tablet_bootstrap.cc:492] T 227a2f18a3a745a2904b3b741d854554 P e003c03f280a41d88a53173792bc21bf: Bootstrap starting.
I20260928 06:33:39.203716 21573 ts_tablet_manager.cc:1428] T 227a2f18a3a745a2904b3b741d854554 P d0163b4acb154ae284884b8295034833: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 06:33:39.203713 21422 heartbeater.cc:503] Master 127.16.157.126:39751 was elected leader, sending a full tablet report...
I20260928 06:33:39.204344 21576 tablet_bootstrap.cc:654] T 227a2f18a3a745a2904b3b741d854554 P e003c03f280a41d88a53173792bc21bf: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:39.204924 21576 log.cc:824] T 227a2f18a3a745a2904b3b741d854554 P e003c03f280a41d88a53173792bc21bf: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:39.205906 21576 tablet_bootstrap.cc:492] T 227a2f18a3a745a2904b3b741d854554 P e003c03f280a41d88a53173792bc21bf: No bootstrap required, opened a new log
I20260928 06:33:39.205992 21576 ts_tablet_manager.cc:1397] T 227a2f18a3a745a2904b3b741d854554 P e003c03f280a41d88a53173792bc21bf: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 06:33:39.207898 21576 raft_consensus.cc:348] T 227a2f18a3a745a2904b3b741d854554 P e003c03f280a41d88a53173792bc21bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35b5a1fc2db54cde806ce230c0f1e472" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 36493 } } peers { permanent_uuid: "d0163b4acb154ae284884b8295034833" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 40147 } } peers { permanent_uuid: "e003c03f280a41d88a53173792bc21bf" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 44809 } }
I20260928 06:33:39.208057 21576 raft_consensus.cc:374] T 227a2f18a3a745a2904b3b741d854554 P e003c03f280a41d88a53173792bc21bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:39.208088 21576 raft_consensus.cc:729] T 227a2f18a3a745a2904b3b741d854554 P e003c03f280a41d88a53173792bc21bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e003c03f280a41d88a53173792bc21bf, State: Initialized, Role: FOLLOWER
I20260928 06:33:39.208189 21576 consensus_queue.cc:260] T 227a2f18a3a745a2904b3b741d854554 P e003c03f280a41d88a53173792bc21bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35b5a1fc2db54cde806ce230c0f1e472" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 36493 } } peers { permanent_uuid: "d0163b4acb154ae284884b8295034833" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 40147 } } peers { permanent_uuid: "e003c03f280a41d88a53173792bc21bf" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 44809 } }
I20260928 06:33:39.208428 21576 ts_tablet_manager.cc:1428] T 227a2f18a3a745a2904b3b741d854554 P e003c03f280a41d88a53173792bc21bf: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 06:33:39.208472 21555 heartbeater.cc:503] Master 127.16.157.126:39751 was elected leader, sending a full tablet report...
I20260928 06:33:39.219472 21580 raft_consensus.cc:482] T 227a2f18a3a745a2904b3b741d854554 P e003c03f280a41d88a53173792bc21bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:33:39.219578 21580 raft_consensus.cc:504] T 227a2f18a3a745a2904b3b741d854554 P e003c03f280a41d88a53173792bc21bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35b5a1fc2db54cde806ce230c0f1e472" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 36493 } } peers { permanent_uuid: "d0163b4acb154ae284884b8295034833" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 40147 } } peers { permanent_uuid: "e003c03f280a41d88a53173792bc21bf" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 44809 } }
I20260928 06:33:39.219887 21580 leader_election.cc:288] T 227a2f18a3a745a2904b3b741d854554 P e003c03f280a41d88a53173792bc21bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35b5a1fc2db54cde806ce230c0f1e472 (127.16.157.65:36493), d0163b4acb154ae284884b8295034833 (127.16.157.66:40147)
I20260928 06:33:39.223194 21557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:39.220036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.65:36493 (local address 127.16.157.67:57407)
0928 06:33:39.220086 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:39.220098 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:39.220146 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:39.220400 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:39.220401 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:39.220402 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:33:39.220464 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:39.220465 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:39.221093 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:39.221094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:39.222230 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:39.222232 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:39.222980 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:39.222981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:39.223068 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:39.223072 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:33:39.223115 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 06:33:39.223143 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 06:33:39.223245 21571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:39.220213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:57407 (local address 127.16.157.65:36493)
0928 06:33:39.220257 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:39.220259 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:39.220266 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:33:39.220290 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:39.220291 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:39.220300 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:39.220350 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:33:39.220520 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:39.221007 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:39.222321 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:39.222887 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:39.222976 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:39.222984 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:39.223141 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 06:33:39.223180 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 06:33:39.223515 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "227a2f18a3a745a2904b3b741d854554" candidate_uuid: "e003c03f280a41d88a53173792bc21bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35b5a1fc2db54cde806ce230c0f1e472" is_pre_election: true
I20260928 06:33:39.223644 21243 raft_consensus.cc:2454] T 227a2f18a3a745a2904b3b741d854554 P 35b5a1fc2db54cde806ce230c0f1e472 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e003c03f280a41d88a53173792bc21bf in term 0.
I20260928 06:33:39.223676 21570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:39.220157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:36691 (local address 127.16.157.66:40147)
0928 06:33:39.220224 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:39.220226 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:39.220307 (+ 81us) server_negotiation.cc:407] Connection header received
0928 06:33:39.220367 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:39.220378 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:39.220388 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:39.220440 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:33:39.220583 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:39.221320 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:39.222529 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:39.222902 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:39.222991 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:39.223000 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:39.223558 (+ 558us) server_negotiation.cc:299] Negotiation successful
0928 06:33:39.223600 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 06:33:39.223886 21581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:39.220036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.66:40147 (local address 127.16.157.67:36691)
0928 06:33:39.220243 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:39.220253 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:39.220314 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:39.220447 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:39.220448 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:39.220450 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:33:39.220523 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:39.220525 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:39.221412 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:39.221413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:39.222448 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:39.222450 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:39.223415 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:39.223415 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:39.223480 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:39.223483 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:33:39.223756 (+ 273us) client_negotiation.cc:240] Negotiation successful
0928 06:33:39.223800 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":10,"thread_start_us":87,"threads_started":1}
I20260928 06:33:39.223905 21445 leader_election.cc:302] T 227a2f18a3a745a2904b3b741d854554 P e003c03f280a41d88a53173792bc21bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35b5a1fc2db54cde806ce230c0f1e472, e003c03f280a41d88a53173792bc21bf; no voters:
I20260928 06:33:39.224043 21580 raft_consensus.cc:2781] T 227a2f18a3a745a2904b3b741d854554 P e003c03f280a41d88a53173792bc21bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:33:39.224105 21580 raft_consensus.cc:482] T 227a2f18a3a745a2904b3b741d854554 P e003c03f280a41d88a53173792bc21bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:33:39.224130 21580 raft_consensus.cc:3037] T 227a2f18a3a745a2904b3b741d854554 P e003c03f280a41d88a53173792bc21bf [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:39.224267 21376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "227a2f18a3a745a2904b3b741d854554" candidate_uuid: "e003c03f280a41d88a53173792bc21bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0163b4acb154ae284884b8295034833" is_pre_election: true
I20260928 06:33:39.224399 21376 raft_consensus.cc:2454] T 227a2f18a3a745a2904b3b741d854554 P d0163b4acb154ae284884b8295034833 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e003c03f280a41d88a53173792bc21bf in term 0.
I20260928 06:33:39.225061 21580 raft_consensus.cc:504] T 227a2f18a3a745a2904b3b741d854554 P e003c03f280a41d88a53173792bc21bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35b5a1fc2db54cde806ce230c0f1e472" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 36493 } } peers { permanent_uuid: "d0163b4acb154ae284884b8295034833" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 40147 } } peers { permanent_uuid: "e003c03f280a41d88a53173792bc21bf" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 44809 } }
I20260928 06:33:39.225188 21580 leader_election.cc:288] T 227a2f18a3a745a2904b3b741d854554 P e003c03f280a41d88a53173792bc21bf [CANDIDATE]: Term 1 election: Requested vote from peers 35b5a1fc2db54cde806ce230c0f1e472 (127.16.157.65:36493), d0163b4acb154ae284884b8295034833 (127.16.157.66:40147)
I20260928 06:33:39.225322 21376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "227a2f18a3a745a2904b3b741d854554" candidate_uuid: "e003c03f280a41d88a53173792bc21bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0163b4acb154ae284884b8295034833"
I20260928 06:33:39.225339 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "227a2f18a3a745a2904b3b741d854554" candidate_uuid: "e003c03f280a41d88a53173792bc21bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35b5a1fc2db54cde806ce230c0f1e472"
I20260928 06:33:39.225404 21243 raft_consensus.cc:3037] T 227a2f18a3a745a2904b3b741d854554 P 35b5a1fc2db54cde806ce230c0f1e472 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:39.225404 21376 raft_consensus.cc:3037] T 227a2f18a3a745a2904b3b741d854554 P d0163b4acb154ae284884b8295034833 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:39.226285 21376 raft_consensus.cc:2454] T 227a2f18a3a745a2904b3b741d854554 P d0163b4acb154ae284884b8295034833 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e003c03f280a41d88a53173792bc21bf in term 1.
I20260928 06:33:39.226285 21243 raft_consensus.cc:2454] T 227a2f18a3a745a2904b3b741d854554 P 35b5a1fc2db54cde806ce230c0f1e472 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e003c03f280a41d88a53173792bc21bf in term 1.
I20260928 06:33:39.226465 21445 leader_election.cc:302] T 227a2f18a3a745a2904b3b741d854554 P e003c03f280a41d88a53173792bc21bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35b5a1fc2db54cde806ce230c0f1e472, e003c03f280a41d88a53173792bc21bf; no voters:
I20260928 06:33:39.226543 21580 raft_consensus.cc:2781] T 227a2f18a3a745a2904b3b741d854554 P e003c03f280a41d88a53173792bc21bf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:33:39.226675 21580 raft_consensus.cc:686] T 227a2f18a3a745a2904b3b741d854554 P e003c03f280a41d88a53173792bc21bf [term 1 LEADER]: Becoming Leader. State: Replica: e003c03f280a41d88a53173792bc21bf, State: Running, Role: LEADER
I20260928 06:33:39.226779 21580 consensus_queue.cc:237] T 227a2f18a3a745a2904b3b741d854554 P e003c03f280a41d88a53173792bc21bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35b5a1fc2db54cde806ce230c0f1e472" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 36493 } } peers { permanent_uuid: "d0163b4acb154ae284884b8295034833" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 40147 } } peers { permanent_uuid: "e003c03f280a41d88a53173792bc21bf" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 44809 } }
I20260928 06:33:39.227530 21099 catalog_manager.cc:5495] T 227a2f18a3a745a2904b3b741d854554 P e003c03f280a41d88a53173792bc21bf reported cstate change: term changed from 0 to 1, leader changed from <none> to e003c03f280a41d88a53173792bc21bf (127.16.157.67). New cstate: current_term: 1 leader_uuid: "e003c03f280a41d88a53173792bc21bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35b5a1fc2db54cde806ce230c0f1e472" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 36493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0163b4acb154ae284884b8295034833" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 40147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e003c03f280a41d88a53173792bc21bf" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 44809 } health_report { overall_health: HEALTHY } } }
W20260928 06:33:39.234157 21423 tablet.cc:2369] T 227a2f18a3a745a2904b3b741d854554 P d0163b4acb154ae284884b8295034833: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:33:39.239384 21569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:39.236505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36100 (local address 127.16.157.67:44809)
0928 06:33:39.236547 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:39.236549 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:39.236560 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:33:39.236611 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:39.236612 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:39.236630 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:39.236705 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:33:39.236835 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:39.237637 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:39.238504 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:39.238810 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:39.238883 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:39.239041 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:33:39.239159 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:33:39.239239 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:39.239291 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 06:33:39.239312 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:33:39.241182 21243 raft_consensus.cc:1261] T 227a2f18a3a745a2904b3b741d854554 P 35b5a1fc2db54cde806ce230c0f1e472 [term 1 FOLLOWER]: Refusing update from remote peer e003c03f280a41d88a53173792bc21bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:33:39.241184 21376 raft_consensus.cc:1261] T 227a2f18a3a745a2904b3b741d854554 P d0163b4acb154ae284884b8295034833 [term 1 FOLLOWER]: Refusing update from remote peer e003c03f280a41d88a53173792bc21bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:33:39.241490 21580 consensus_queue.cc:1035] T 227a2f18a3a745a2904b3b741d854554 P e003c03f280a41d88a53173792bc21bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "35b5a1fc2db54cde806ce230c0f1e472" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 36493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:33:39.241606 21582 consensus_queue.cc:1035] T 227a2f18a3a745a2904b3b741d854554 P e003c03f280a41d88a53173792bc21bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0163b4acb154ae284884b8295034833" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 40147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:33:39.243994 21585 mvcc.cc:205] Tried to move back new op lower bound from 7334204290009473024 to 7334204289953738752. Current Snapshot: MvccSnapshot[applied={T|T < 7334204290009473024}]
I20260928 06:33:39.244217 21586 mvcc.cc:205] Tried to move back new op lower bound from 7334204290009473024 to 7334204289953738752. Current Snapshot: MvccSnapshot[applied={T|T < 7334204290009473024}]
I20260928 06:33:39.247215 21098 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:33:39.247313 21098 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:33:39.248157 21098 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 79bf49e53bb24800b20a6984d9124269: Sending DeleteTablet for 3 replicas of tablet 227a2f18a3a745a2904b3b741d854554
I20260928 06:33:39.248522 21356 tablet_service.cc:1514] Processing DeleteTablet for tablet 227a2f18a3a745a2904b3b741d854554 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:33:39 UTC) from {username='kudu', principal='kudu/127.16.157.126@KRBTEST.COM'} at 127.0.0.1:46260
I20260928 06:33:39.248544 21223 tablet_service.cc:1514] Processing DeleteTablet for tablet 227a2f18a3a745a2904b3b741d854554 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:33:39 UTC) from {username='kudu', principal='kudu/127.16.157.126@KRBTEST.COM'} at 127.0.0.1:59846
I20260928 06:33:39.248764 21594 tablet_replica.cc:330] T 227a2f18a3a745a2904b3b741d854554 P d0163b4acb154ae284884b8295034833: stopping tablet replica
I20260928 06:33:39.248867 21594 raft_consensus.cc:2229] T 227a2f18a3a745a2904b3b741d854554 P d0163b4acb154ae284884b8295034833 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:33:39.248868 21489 tablet_service.cc:1514] Processing DeleteTablet for tablet 227a2f18a3a745a2904b3b741d854554 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:33:39 UTC) from {username='kudu', principal='kudu/127.16.157.126@KRBTEST.COM'} at 127.0.0.1:36098
I20260928 06:33:39.248973 21594 raft_consensus.cc:2258] T 227a2f18a3a745a2904b3b741d854554 P d0163b4acb154ae284884b8295034833 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:33:39.249130 21595 tablet_replica.cc:330] T 227a2f18a3a745a2904b3b741d854554 P 35b5a1fc2db54cde806ce230c0f1e472: stopping tablet replica
I20260928 06:33:39.249228 21595 raft_consensus.cc:2229] T 227a2f18a3a745a2904b3b741d854554 P 35b5a1fc2db54cde806ce230c0f1e472 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:33:39.249235 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 21161
I20260928 06:33:39.249325 21595 raft_consensus.cc:2258] T 227a2f18a3a745a2904b3b741d854554 P 35b5a1fc2db54cde806ce230c0f1e472 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:33:39.249805 21596 tablet_replica.cc:330] T 227a2f18a3a745a2904b3b741d854554 P e003c03f280a41d88a53173792bc21bf: stopping tablet replica
I20260928 06:33:39.250555 21594 ts_tablet_manager.cc:1905] T 227a2f18a3a745a2904b3b741d854554 P d0163b4acb154ae284884b8295034833: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:33:39.251783 21596 raft_consensus.cc:2229] T 227a2f18a3a745a2904b3b741d854554 P e003c03f280a41d88a53173792bc21bf [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:33:39.251935 21596 raft_consensus.cc:2258] T 227a2f18a3a745a2904b3b741d854554 P e003c03f280a41d88a53173792bc21bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:33:39.252422 21596 ts_tablet_manager.cc:1905] T 227a2f18a3a745a2904b3b741d854554 P e003c03f280a41d88a53173792bc21bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:33:39.254047 21596 ts_tablet_manager.cc:1918] T 227a2f18a3a745a2904b3b741d854554 P e003c03f280a41d88a53173792bc21bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:33:39.254168 21596 log.cc:1196] T 227a2f18a3a745a2904b3b741d854554 P e003c03f280a41d88a53173792bc21bf: Deleting WAL directory at /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577196946045-17013-0/minicluster-data/ts-2/wal/wals/227a2f18a3a745a2904b3b741d854554
I20260928 06:33:39.254518 21596 ts_tablet_manager.cc:1939] T 227a2f18a3a745a2904b3b741d854554 P e003c03f280a41d88a53173792bc21bf: Deleting consensus metadata
I20260928 06:33:39.254947 21085 catalog_manager.cc:4841] TS e003c03f280a41d88a53173792bc21bf (127.16.157.67:44809): tablet 227a2f18a3a745a2904b3b741d854554 (table test-table [id=247d84bcfa5c4d2b99e004570c4146bc]) successfully deleted
W20260928 06:33:39.257671 21086 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.157.65:36493 (error 108)
I20260928 06:33:39.257967 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 21294
I20260928 06:33:39.261982 21567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:39.261810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.65:36493 (local address 127.0.0.1:59852)
0928 06:33:39.261882 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:39.261929 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.65:36493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 06:33:39.265081 21086 catalog_manager.cc:4584] TS 35b5a1fc2db54cde806ce230c0f1e472 (127.16.157.65:36493): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 227a2f18a3a745a2904b3b741d854554: Network error: Client connection negotiation failed: client connection to 127.16.157.65:36493: connect: Connection refused (error 111)
I20260928 06:33:39.265271 21567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:39.265169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.66:40147 (local address 127.0.0.1:46262)
0928 06:33:39.265217 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:39.265245 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.66:40147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 06:33:39.265329 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 21427
I20260928 06:33:39.270320 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 21068
2026-09-28T06:33:39Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2003 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T06:33:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:33:39Z Disabled control of system clock
I20260928 06:33:39.300089 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577196946045-17013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577196946045-17013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577196946045-17013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:33073
--webserver_interface=127.16.157.126
--webserver_port=0
--builtin_ntp_servers=127.16.157.84:35333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.157.126:33073
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 06:33:39.393257 21601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:39.393458 21601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:39.393478 21601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:39.394671 21601 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:33:39.394711 21601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:39.394723 21601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:39.394778 21601 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:33:39.394825 21601 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:33:39.396157 21601 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:35333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577196946045-17013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577196946045-17013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.157.126:33073
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:33073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21601
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577196946045-17013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:39.396396 21601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:39.396705 21601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:39.401310 21609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:39.401268 21606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:39.401266 21607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:39.401701 21601 server_base.cc:1034] running on GCE node
I20260928 06:33:39.401934 21601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:39.402247 21601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:39.403401 21601 hybrid_clock.cc:648] HybridClock initialized: now 1790577219403387 us; error 35 us; skew 500 ppm
I20260928 06:33:39.404764 21601 webserver.cc:460] Webserver started at http://127.16.157.126:37959/ using document root <none> and password file <none>
I20260928 06:33:39.405014 21601 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:39.405057 21601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:39.405192 21601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:39.406279 21601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577196946045-17013-0/minicluster-data/master-0/data/instance:
uuid: "c271cf65405b44e69bcef1b1bd71eb0b"
format_stamp: "Formatted at 2026-09-28 06:33:39 on dist-test-slave-rrj5"
server_key: "cda34b7d7c4c44c427e8d6b62a579188"
server_key_iv: "c193fcb715e288d9335d9af854de5f3a"
server_key_version: "encryptionkey@0"
I20260928 06:33:39.406626 21601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577196946045-17013-0/minicluster-data/master-0/wal/instance:
uuid: "c271cf65405b44e69bcef1b1bd71eb0b"
format_stamp: "Formatted at 2026-09-28 06:33:39 on dist-test-slave-rrj5"
server_key: "cda34b7d7c4c44c427e8d6b62a579188"
server_key_iv: "c193fcb715e288d9335d9af854de5f3a"
server_key_version: "encryptionkey@0"
I20260928 06:33:39.407917 21601 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:33:39.408658 21615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:39.408813 21601 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:33:39.408911 21601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577196946045-17013-0/minicluster-data/master-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577196946045-17013-0/minicluster-data/master-0/wal
uuid: "c271cf65405b44e69bcef1b1bd71eb0b"
format_stamp: "Formatted at 2026-09-28 06:33:39 on dist-test-slave-rrj5"
server_key: "cda34b7d7c4c44c427e8d6b62a579188"
server_key_iv: "c193fcb715e288d9335d9af854de5f3a"
server_key_version: "encryptionkey@0"
I20260928 06:33:39.408978 21601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577196946045-17013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577196946045-17013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577196946045-17013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:39.412307 21601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:39.412567 21601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:39.412695 21601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:39.417184 21601 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.126:33073
I20260928 06:33:39.417212 21667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.126:33073 every 8 connection(s)
I20260928 06:33:39.417514 21601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
I20260928 06:33:39.418181 21668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:39.420750 21668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c271cf65405b44e69bcef1b1bd71eb0b: Bootstrap starting.
I20260928 06:33:39.421311 21668 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c271cf65405b44e69bcef1b1bd71eb0b: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:39.421649 21668 log.cc:824] T 00000000000000000000000000000000 P c271cf65405b44e69bcef1b1bd71eb0b: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:39.422363 21668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c271cf65405b44e69bcef1b1bd71eb0b: No bootstrap required, opened a new log
I20260928 06:33:39.423842 21668 raft_consensus.cc:348] T 00000000000000000000000000000000 P c271cf65405b44e69bcef1b1bd71eb0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c271cf65405b44e69bcef1b1bd71eb0b" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 33073 } }
I20260928 06:33:39.423969 21668 raft_consensus.cc:374] T 00000000000000000000000000000000 P c271cf65405b44e69bcef1b1bd71eb0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:39.423990 21668 raft_consensus.cc:729] T 00000000000000000000000000000000 P c271cf65405b44e69bcef1b1bd71eb0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c271cf65405b44e69bcef1b1bd71eb0b, State: Initialized, Role: FOLLOWER
I20260928 06:33:39.424132 21668 consensus_queue.cc:260] T 00000000000000000000000000000000 P c271cf65405b44e69bcef1b1bd71eb0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c271cf65405b44e69bcef1b1bd71eb0b" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 33073 } }
I20260928 06:33:39.424201 21668 raft_consensus.cc:388] T 00000000000000000000000000000000 P c271cf65405b44e69bcef1b1bd71eb0b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:33:39.424250 21668 raft_consensus.cc:482] T 00000000000000000000000000000000 P c271cf65405b44e69bcef1b1bd71eb0b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:33:39.424304 21668 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c271cf65405b44e69bcef1b1bd71eb0b [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:39.424803 21668 raft_consensus.cc:504] T 00000000000000000000000000000000 P c271cf65405b44e69bcef1b1bd71eb0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c271cf65405b44e69bcef1b1bd71eb0b" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 33073 } }
I20260928 06:33:39.424883 21668 leader_election.cc:302] T 00000000000000000000000000000000 P c271cf65405b44e69bcef1b1bd71eb0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c271cf65405b44e69bcef1b1bd71eb0b; no voters:
I20260928 06:33:39.425022 21668 leader_election.cc:288] T 00000000000000000000000000000000 P c271cf65405b44e69bcef1b1bd71eb0b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:33:39.425035 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 21601
I20260928 06:33:39.425053 21671 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c271cf65405b44e69bcef1b1bd71eb0b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:33:39.425117 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577196946045-17013-0/minicluster-data/master-0/wal/instance
I20260928 06:33:39.425173 21671 raft_consensus.cc:686] T 00000000000000000000000000000000 P c271cf65405b44e69bcef1b1bd71eb0b [term 1 LEADER]: Becoming Leader. State: Replica: c271cf65405b44e69bcef1b1bd71eb0b, State: Running, Role: LEADER
I20260928 06:33:39.425252 21668 sys_catalog.cc:564] T 00000000000000000000000000000000 P c271cf65405b44e69bcef1b1bd71eb0b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:33:39.425275 17013 external_mini_cluster.cc:1428] Setting key e789615756666eee0dc2fc9c007dbba2
I20260928 06:33:39.425239 21671 consensus_queue.cc:237] T 00000000000000000000000000000000 P c271cf65405b44e69bcef1b1bd71eb0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c271cf65405b44e69bcef1b1bd71eb0b" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 33073 } }
I20260928 06:33:39.425539 21671 sys_catalog.cc:455] T 00000000000000000000000000000000 P c271cf65405b44e69bcef1b1bd71eb0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c271cf65405b44e69bcef1b1bd71eb0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c271cf65405b44e69bcef1b1bd71eb0b" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 33073 } } }
I20260928 06:33:39.425663 21671 sys_catalog.cc:458] T 00000000000000000000000000000000 P c271cf65405b44e69bcef1b1bd71eb0b [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:39.425819 21671 sys_catalog.cc:455] T 00000000000000000000000000000000 P c271cf65405b44e69bcef1b1bd71eb0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader c271cf65405b44e69bcef1b1bd71eb0b. Latest consensus state: current_term: 1 leader_uuid: "c271cf65405b44e69bcef1b1bd71eb0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c271cf65405b44e69bcef1b1bd71eb0b" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 33073 } } }
I20260928 06:33:39.425886 21671 sys_catalog.cc:458] T 00000000000000000000000000000000 P c271cf65405b44e69bcef1b1bd71eb0b [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:39.426554 21675 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:33:39.427073 21675 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:33:39.428808 21675 catalog_manager.cc:1347] Generated new cluster ID: 92fd160509ee46d2a1d23d338430b1c3
I20260928 06:33:39.428866 21675 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:33:39.429504 21676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:39.425745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51664 (local address 127.16.157.126:33073)
0928 06:33:39.426019 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:39.426028 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:39.427597 (+ 1569us) server_negotiation.cc:407] Connection header received
0928 06:33:39.427796 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:39.427804 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:39.427854 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:39.427973 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:39.429057 (+ 1084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:39.429060 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:39.429060 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:33:39.429065 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:39.429109 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:33:39.429122 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:39.429153 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:39.429272 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 06:33:39.429323 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":65,"threads_started":1}
I20260928 06:33:39.439181 21675 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:33:39.439357 21675 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:33:39.444634 21675 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c271cf65405b44e69bcef1b1bd71eb0b: Generated new TSK 0
I20260928 06:33:39.444840 21675 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:33:39.446604 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577196946045-17013-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577196946045-17013-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577196946045-17013-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.65:0
--local_ip_for_outbound_sockets=127.16.157.65
--webserver_interface=127.16.157.65
--webserver_port=0
--tserver_master_addrs=127.16.157.126:33073
--builtin_ntp_servers=127.16.157.84:35333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 06:33:39.531041 21692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:39.531253 21692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:39.531289 21692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:39.532398 21692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:39.532470 21692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:39.532512 21692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.65
I20260928 06:33:39.533911 21692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:35333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577196946045-17013-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577196946045-17013-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.157.65
--webserver_port=0
--tserver_master_addrs=127.16.157.126:33073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.65
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577196946045-17013-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:39.534147 21692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:39.534403 21692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:39.539134 21700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:39.539160 21697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:39.539399 21698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:39.539609 21692 server_base.cc:1034] running on GCE node
I20260928 06:33:39.539808 21692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:39.540063 21692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:39.541226 21692 hybrid_clock.cc:648] HybridClock initialized: now 1790577219541217 us; error 26 us; skew 500 ppm
I20260928 06:33:39.542611 21692 webserver.cc:460] Webserver started at http://127.16.157.65:37985/ using document root <none> and password file <none>
I20260928 06:33:39.542868 21692 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:39.542936 21692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:39.543069 21692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:39.544076 21692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577196946045-17013-0/minicluster-data/ts-0/data/instance:
uuid: "36197995c3dd478eba3cf60405bc61d8"
format_stamp: "Formatted at 2026-09-28 06:33:39 on dist-test-slave-rrj5"
server_key: "130ccfe348655955392e9ddebea5f401"
server_key_iv: "fc763ccad36350508ae1e71f0cbd8dc8"
server_key_version: "encryptionkey@0"
I20260928 06:33:39.544446 21692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577196946045-17013-0/minicluster-data/ts-0/wal/instance:
uuid: "36197995c3dd478eba3cf60405bc61d8"
format_stamp: "Formatted at 2026-09-28 06:33:39 on dist-test-slave-rrj5"
server_key: "130ccfe348655955392e9ddebea5f401"
server_key_iv: "fc763ccad36350508ae1e71f0cbd8dc8"
server_key_version: "encryptionkey@0"
I20260928 06:33:39.545810 21692 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:33:39.546579 21706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:39.546746 21692 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:39.546828 21692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577196946045-17013-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577196946045-17013-0/minicluster-data/ts-0/wal
uuid: "36197995c3dd478eba3cf60405bc61d8"
format_stamp: "Formatted at 2026-09-28 06:33:39 on dist-test-slave-rrj5"
server_key: "130ccfe348655955392e9ddebea5f401"
server_key_iv: "fc763ccad36350508ae1e71f0cbd8dc8"
server_key_version: "encryptionkey@0"
I20260928 06:33:39.546895 21692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577196946045-17013-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577196946045-17013-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577196946045-17013-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:39.550419 21692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:39.550691 21692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:39.550843 21692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:39.551083 21692 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:33:39.551424 21692 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:33:39.551482 21692 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:39.551518 21692 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:33:39.551543 21692 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:39.558583 21692 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.65:33451
I20260928 06:33:39.558616 21819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.65:33451 every 8 connection(s)
I20260928 06:33:39.558991 21692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
I20260928 06:33:39.562448 21676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:39.559793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:52079 (local address 127.16.157.126:33073)
0928 06:33:39.559841 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:39.559843 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:39.560147 (+ 304us) server_negotiation.cc:407] Connection header received
0928 06:33:39.560765 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:39.560766 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:39.560772 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:39.560818 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:39.562160 (+ 1342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:39.562161 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:39.562161 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:33:39.562164 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:39.562177 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:33:39.562182 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:39.562250 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:39.562363 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 06:33:39.562380 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 06:33:39.562768 21822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:39.559791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:33073 (local address 127.16.157.65:52079)
0928 06:33:39.559994 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:39.560042 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:39.560659 (+ 617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:39.560892 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:39.560896 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:39.560905 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:39.561986 (+ 1081us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:33:39.561986 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:39.562025 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:33:39.562060 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:39.562265 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:39.562271 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:39.562274 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:33:39.562361 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 06:33:39.562423 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":83,"threads_started":1}
I20260928 06:33:39.563314 21820 heartbeater.cc:347] Connected to a master server at 127.16.157.126:33073
I20260928 06:33:39.563426 21820 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:39.563573 21820 heartbeater.cc:511] Master 127.16.157.126:33073 requested a full tablet report, sending...
I20260928 06:33:39.564114 21632 ts_manager.cc:194] Registered new tserver with Master: 36197995c3dd478eba3cf60405bc61d8 (127.16.157.65:33451)
I20260928 06:33:39.566247 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 21692
I20260928 06:33:39.566351 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577196946045-17013-0/minicluster-data/ts-0/wal/instance
I20260928 06:33:39.566526 17013 external_mini_cluster.cc:1428] Setting key 3926e5c9624f737f1304b7f4948fde2b
I20260928 06:33:39.567451 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577196946045-17013-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577196946045-17013-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577196946045-17013-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.66:0
--local_ip_for_outbound_sockets=127.16.157.66
--webserver_interface=127.16.157.66
--webserver_port=0
--tserver_master_addrs=127.16.157.126:33073
--builtin_ntp_servers=127.16.157.84:35333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 06:33:39.663973 21823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:39.664204 21823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:39.664240 21823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:39.665375 21823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:39.665436 21823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:39.665475 21823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.66
I20260928 06:33:39.666878 21823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:35333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577196946045-17013-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577196946045-17013-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.157.66
--webserver_port=0
--tserver_master_addrs=127.16.157.126:33073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.66
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577196946045-17013-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:39.667155 21823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:39.667443 21823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:39.672369 21828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:39.672369 21829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:39.672576 21823 server_base.cc:1034] running on GCE node
W20260928 06:33:39.672405 21831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:39.672917 21823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:39.673197 21823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:39.674362 21823 hybrid_clock.cc:648] HybridClock initialized: now 1790577219674353 us; error 32 us; skew 500 ppm
I20260928 06:33:39.675624 21823 webserver.cc:460] Webserver started at http://127.16.157.66:40059/ using document root <none> and password file <none>
I20260928 06:33:39.675876 21823 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:39.675948 21823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:39.676103 21823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:39.677201 21823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577196946045-17013-0/minicluster-data/ts-1/data/instance:
uuid: "1373fae1f2c24e6b965739e1fa390175"
format_stamp: "Formatted at 2026-09-28 06:33:39 on dist-test-slave-rrj5"
server_key: "2f7f96bc42d318a3650a76c5ec322eb9"
server_key_iv: "36282bfed3af5200df61c2514c9e8f79"
server_key_version: "encryptionkey@0"
I20260928 06:33:39.677608 21823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577196946045-17013-0/minicluster-data/ts-1/wal/instance:
uuid: "1373fae1f2c24e6b965739e1fa390175"
format_stamp: "Formatted at 2026-09-28 06:33:39 on dist-test-slave-rrj5"
server_key: "2f7f96bc42d318a3650a76c5ec322eb9"
server_key_iv: "36282bfed3af5200df61c2514c9e8f79"
server_key_version: "encryptionkey@0"
I20260928 06:33:39.678979 21823 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:39.679785 21837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:39.679965 21823 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:39.680056 21823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577196946045-17013-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577196946045-17013-0/minicluster-data/ts-1/wal
uuid: "1373fae1f2c24e6b965739e1fa390175"
format_stamp: "Formatted at 2026-09-28 06:33:39 on dist-test-slave-rrj5"
server_key: "2f7f96bc42d318a3650a76c5ec322eb9"
server_key_iv: "36282bfed3af5200df61c2514c9e8f79"
server_key_version: "encryptionkey@0"
I20260928 06:33:39.680153 21823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577196946045-17013-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577196946045-17013-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577196946045-17013-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:39.683504 21823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:39.683768 21823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:39.683918 21823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:39.684122 21823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:33:39.684453 21823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:33:39.684514 21823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:39.684553 21823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:33:39.684579 21823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:39.691403 21823 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.66:36359
I20260928 06:33:39.691433 21950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.66:36359 every 8 connection(s)
I20260928 06:33:39.691763 21823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
I20260928 06:33:39.694882 21676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:39.692483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.66:53921 (local address 127.16.157.126:33073)
0928 06:33:39.692537 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:39.692540 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:39.692783 (+ 243us) server_negotiation.cc:407] Connection header received
0928 06:33:39.693278 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:39.693280 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:39.693287 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:39.693371 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:39.694628 (+ 1257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:39.694629 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:39.694630 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:33:39.694635 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:39.694655 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:33:39.694660 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:39.694744 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:39.694799 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 06:33:39.694815 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 06:33:39.695046 21953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:39.692471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:33073 (local address 127.16.157.66:53921)
0928 06:33:39.692662 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:39.692704 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:39.693169 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:39.693439 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:39.693442 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:39.693451 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:39.694470 (+ 1019us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:33:39.694471 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:39.694499 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:33:39.694527 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:39.694739 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:39.694741 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:39.694744 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:33:39.694793 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 06:33:39.694840 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":85,"threads_started":1}
I20260928 06:33:39.695551 21951 heartbeater.cc:347] Connected to a master server at 127.16.157.126:33073
I20260928 06:33:39.695662 21951 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:39.695806 21951 heartbeater.cc:511] Master 127.16.157.126:33073 requested a full tablet report, sending...
I20260928 06:33:39.696202 21632 ts_manager.cc:194] Registered new tserver with Master: 1373fae1f2c24e6b965739e1fa390175 (127.16.157.66:36359)
I20260928 06:33:39.696724 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 21823
I20260928 06:33:39.696815 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577196946045-17013-0/minicluster-data/ts-1/wal/instance
I20260928 06:33:39.696979 17013 external_mini_cluster.cc:1428] Setting key 0555bc9668f932894f205cefc6180493
I20260928 06:33:39.697854 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577196946045-17013-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577196946045-17013-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577196946045-17013-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.67:0
--local_ip_for_outbound_sockets=127.16.157.67
--webserver_interface=127.16.157.67
--webserver_port=0
--tserver_master_addrs=127.16.157.126:33073
--builtin_ntp_servers=127.16.157.84:35333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 06:33:39.777849 21954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:39.778040 21954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:39.778062 21954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:39.779207 21954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:39.779245 21954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:39.779270 21954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.67
I20260928 06:33:39.780592 21954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:35333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577196946045-17013-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577196946045-17013-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.157.67
--webserver_port=0
--tserver_master_addrs=127.16.157.126:33073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.67
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577196946045-17013-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:39.780824 21954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:39.781126 21954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:39.786211 21959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:39.786216 21960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:39.786573 21954 server_base.cc:1034] running on GCE node
W20260928 06:33:39.786273 21962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:39.786832 21954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:39.787072 21954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:39.788235 21954 hybrid_clock.cc:648] HybridClock initialized: now 1790577219788213 us; error 41 us; skew 500 ppm
I20260928 06:33:39.789485 21954 webserver.cc:460] Webserver started at http://127.16.157.67:39007/ using document root <none> and password file <none>
I20260928 06:33:39.789793 21954 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:39.789875 21954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:39.790020 21954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:39.791049 21954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577196946045-17013-0/minicluster-data/ts-2/data/instance:
uuid: "f6e3549d46d2493c900331c107cea0bd"
format_stamp: "Formatted at 2026-09-28 06:33:39 on dist-test-slave-rrj5"
server_key: "8e7d562c852d93f0eb89347f2044affe"
server_key_iv: "bc322bf0ca75c59eea20867064c39868"
server_key_version: "encryptionkey@0"
I20260928 06:33:39.791409 21954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577196946045-17013-0/minicluster-data/ts-2/wal/instance:
uuid: "f6e3549d46d2493c900331c107cea0bd"
format_stamp: "Formatted at 2026-09-28 06:33:39 on dist-test-slave-rrj5"
server_key: "8e7d562c852d93f0eb89347f2044affe"
server_key_iv: "bc322bf0ca75c59eea20867064c39868"
server_key_version: "encryptionkey@0"
I20260928 06:33:39.792797 21954 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:33:39.793547 21968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:39.793722 21954 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:33:39.793820 21954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577196946045-17013-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577196946045-17013-0/minicluster-data/ts-2/wal
uuid: "f6e3549d46d2493c900331c107cea0bd"
format_stamp: "Formatted at 2026-09-28 06:33:39 on dist-test-slave-rrj5"
server_key: "8e7d562c852d93f0eb89347f2044affe"
server_key_iv: "bc322bf0ca75c59eea20867064c39868"
server_key_version: "encryptionkey@0"
I20260928 06:33:39.793902 21954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577196946045-17013-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577196946045-17013-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577196946045-17013-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:39.797115 21954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:39.797384 21954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:39.797523 21954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:39.797793 21954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:33:39.798138 21954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:33:39.798192 21954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:39.798228 21954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:33:39.798254 21954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:39.805282 21954 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.67:36527
I20260928 06:33:39.805316 22082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.67:36527 every 8 connection(s)
I20260928 06:33:39.805747 21954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
I20260928 06:33:39.808694 21676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:39.806501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:58647 (local address 127.16.157.126:33073)
0928 06:33:39.806566 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:39.806569 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:39.806769 (+ 200us) server_negotiation.cc:407] Connection header received
0928 06:33:39.807216 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:39.807217 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:39.807223 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:39.807304 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:39.808449 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:39.808450 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:39.808451 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:33:39.808455 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:39.808470 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:33:39.808475 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:39.808540 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:39.808604 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 06:33:39.808619 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 06:33:39.808828 22085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:39.806464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:33073 (local address 127.16.157.67:58647)
0928 06:33:39.806642 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:39.806679 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:39.807128 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:39.807352 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:39.807355 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:39.807363 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:39.808311 (+ 948us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:33:39.808312 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:39.808339 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:33:39.808365 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:39.808537 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:39.808539 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:39.808542 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:33:39.808594 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 06:33:39.808637 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":96,"threads_started":1}
I20260928 06:33:39.809337 22083 heartbeater.cc:347] Connected to a master server at 127.16.157.126:33073
I20260928 06:33:39.809440 22083 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:39.809607 22083 heartbeater.cc:511] Master 127.16.157.126:33073 requested a full tablet report, sending...
I20260928 06:33:39.809964 21632 ts_manager.cc:194] Registered new tserver with Master: f6e3549d46d2493c900331c107cea0bd (127.16.157.67:36527)
I20260928 06:33:39.812610 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 21954
I20260928 06:33:39.812685 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577196946045-17013-0/minicluster-data/ts-2/wal/instance
I20260928 06:33:39.812863 17013 external_mini_cluster.cc:1428] Setting key a4577c06af07b9dac1a31e550a6e85d4
I20260928 06:33:39.813717 17013 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:33:39.816083 21676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:39.815031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51680 (local address 127.16.157.126:33073)
0928 06:33:39.815072 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:39.815074 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:39.815194 (+ 120us) server_negotiation.cc:407] Connection header received
0928 06:33:39.815254 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:39.815255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:39.815259 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:39.815314 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:39.815903 (+ 589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:39.815904 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:39.815904 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:33:39.815907 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:39.815922 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:33:39.815925 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:39.815980 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:39.816011 (+ 31us) server_negotiation.cc:299] Negotiation successful
0928 06:33:39.816024 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 06:33:39.817764 21632 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51680:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:33:39.819247 21632 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:33:39.822948 22093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:39.821170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.67:36527 (local address 127.0.0.1:46092)
0928 06:33:39.821523 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:39.821539 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:39.821667 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:39.822109 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:39.822110 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:39.822112 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:39.822521 (+ 409us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:33:39.822521 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:39.822537 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:33:39.822561 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:39.822790 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:39.822796 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:39.822799 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:33:39.822860 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 06:33:39.822874 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":92,"threads_started":1}
I20260928 06:33:39.823239 22094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:39.821367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46092 (local address 127.16.157.67:36527)
0928 06:33:39.821528 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:39.821533 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:39.821642 (+ 109us) server_negotiation.cc:407] Connection header received
0928 06:33:39.821737 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:39.821740 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:39.821747 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:39.821814 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:39.822645 (+ 831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:39.822651 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:39.822651 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:33:39.822656 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:39.822690 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:33:39.822709 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:39.822762 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:39.823145 (+ 383us) server_negotiation.cc:299] Negotiation successful
0928 06:33:39.823162 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":86,"threads_started":1}
I20260928 06:33:39.823256 22091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:39.821331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.66:36359 (local address 127.0.0.1:36740)
0928 06:33:39.821964 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:39.821974 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:39.822024 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:39.822427 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:39.822428 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:39.822430 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:39.822798 (+ 368us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:33:39.822798 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:39.822805 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:33:39.822816 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:39.823143 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:39.823144 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:39.823145 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 06:33:39.823183 (+ 38us) client_negotiation.cc:240] Negotiation successful
0928 06:33:39.823195 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":62,"threads_started":1}
I20260928 06:33:39.823566 22096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:39.821749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36740 (local address 127.16.157.66:36359)
0928 06:33:39.821913 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:39.821916 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:39.822074 (+ 158us) server_negotiation.cc:407] Connection header received
0928 06:33:39.822099 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:39.822101 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:39.822109 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:39.822171 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:39.822899 (+ 728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:39.822907 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:39.822907 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:33:39.822912 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:39.822944 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:33:39.822967 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:39.823015 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:39.823475 (+ 460us) server_negotiation.cc:299] Negotiation successful
0928 06:33:39.823491 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":76,"threads_started":1}
I20260928 06:33:39.823904 22092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:39.821167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.65:33451 (local address 127.0.0.1:53198)
0928 06:33:39.821314 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:39.821353 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:39.821411 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:39.821838 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:39.821843 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:39.821849 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:39.823095 (+ 1246us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:33:39.823096 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:39.823102 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:33:39.823114 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:39.823791 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:39.823792 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:39.823793 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 06:33:39.823823 (+ 30us) client_negotiation.cc:240] Negotiation successful
0928 06:33:39.823835 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":49,"threads_started":1}
I20260928 06:33:39.823856 22016 tablet_service.cc:1467] Processing CreateTablet for tablet be06870d950d419eb09d8557a45830dd (DEFAULT_TABLE table=test-table [id=0a213699963f41c28efdd03a0a732e62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:33:39.824208 22095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:39.821462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53198 (local address 127.16.157.65:33451)
0928 06:33:39.821684 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:39.821688 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:39.821698 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:33:39.821724 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:39.821727 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:39.821736 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:39.823189 (+ 1453us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:39.823586 (+ 397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:39.823589 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:39.823589 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:33:39.823592 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:39.823691 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:33:39.823707 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:39.824078 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:39.824103 (+ 25us) server_negotiation.cc:299] Negotiation successful
0928 06:33:39.824118 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":118,"threads_started":1}
I20260928 06:33:39.824113 21885 tablet_service.cc:1467] Processing CreateTablet for tablet be06870d950d419eb09d8557a45830dd (DEFAULT_TABLE table=test-table [id=0a213699963f41c28efdd03a0a732e62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:33:39.824216 22016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be06870d950d419eb09d8557a45830dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:39.824410 21885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be06870d950d419eb09d8557a45830dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:39.826300 21754 tablet_service.cc:1467] Processing CreateTablet for tablet be06870d950d419eb09d8557a45830dd (DEFAULT_TABLE table=test-table [id=0a213699963f41c28efdd03a0a732e62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:33:39.826627 21754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be06870d950d419eb09d8557a45830dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:39.828110 22098 tablet_bootstrap.cc:492] T be06870d950d419eb09d8557a45830dd P 1373fae1f2c24e6b965739e1fa390175: Bootstrap starting.
I20260928 06:33:39.828331 22097 tablet_bootstrap.cc:492] T be06870d950d419eb09d8557a45830dd P f6e3549d46d2493c900331c107cea0bd: Bootstrap starting.
I20260928 06:33:39.828751 22098 tablet_bootstrap.cc:654] T be06870d950d419eb09d8557a45830dd P 1373fae1f2c24e6b965739e1fa390175: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:39.828931 22097 tablet_bootstrap.cc:654] T be06870d950d419eb09d8557a45830dd P f6e3549d46d2493c900331c107cea0bd: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:39.829046 22098 log.cc:824] T be06870d950d419eb09d8557a45830dd P 1373fae1f2c24e6b965739e1fa390175: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:39.829216 22097 log.cc:824] T be06870d950d419eb09d8557a45830dd P f6e3549d46d2493c900331c107cea0bd: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:39.829938 22098 tablet_bootstrap.cc:492] T be06870d950d419eb09d8557a45830dd P 1373fae1f2c24e6b965739e1fa390175: No bootstrap required, opened a new log
I20260928 06:33:39.829979 22097 tablet_bootstrap.cc:492] T be06870d950d419eb09d8557a45830dd P f6e3549d46d2493c900331c107cea0bd: No bootstrap required, opened a new log
I20260928 06:33:39.830005 22098 ts_tablet_manager.cc:1397] T be06870d950d419eb09d8557a45830dd P 1373fae1f2c24e6b965739e1fa390175: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:33:39.830067 22097 ts_tablet_manager.cc:1397] T be06870d950d419eb09d8557a45830dd P f6e3549d46d2493c900331c107cea0bd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:33:39.831866 22097 raft_consensus.cc:348] T be06870d950d419eb09d8557a45830dd P f6e3549d46d2493c900331c107cea0bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36197995c3dd478eba3cf60405bc61d8" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 33451 } } peers { permanent_uuid: "1373fae1f2c24e6b965739e1fa390175" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 36359 } } peers { permanent_uuid: "f6e3549d46d2493c900331c107cea0bd" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 36527 } }
I20260928 06:33:39.832023 22097 raft_consensus.cc:374] T be06870d950d419eb09d8557a45830dd P f6e3549d46d2493c900331c107cea0bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:39.832073 22097 raft_consensus.cc:729] T be06870d950d419eb09d8557a45830dd P f6e3549d46d2493c900331c107cea0bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6e3549d46d2493c900331c107cea0bd, State: Initialized, Role: FOLLOWER
I20260928 06:33:39.832181 22097 consensus_queue.cc:260] T be06870d950d419eb09d8557a45830dd P f6e3549d46d2493c900331c107cea0bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36197995c3dd478eba3cf60405bc61d8" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 33451 } } peers { permanent_uuid: "1373fae1f2c24e6b965739e1fa390175" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 36359 } } peers { permanent_uuid: "f6e3549d46d2493c900331c107cea0bd" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 36527 } }
I20260928 06:33:39.832487 22097 ts_tablet_manager.cc:1428] T be06870d950d419eb09d8557a45830dd P f6e3549d46d2493c900331c107cea0bd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:33:39.832497 22083 heartbeater.cc:503] Master 127.16.157.126:33073 was elected leader, sending a full tablet report...
I20260928 06:33:39.832701 22098 raft_consensus.cc:348] T be06870d950d419eb09d8557a45830dd P 1373fae1f2c24e6b965739e1fa390175 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36197995c3dd478eba3cf60405bc61d8" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 33451 } } peers { permanent_uuid: "1373fae1f2c24e6b965739e1fa390175" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 36359 } } peers { permanent_uuid: "f6e3549d46d2493c900331c107cea0bd" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 36527 } }
I20260928 06:33:39.832858 22098 raft_consensus.cc:374] T be06870d950d419eb09d8557a45830dd P 1373fae1f2c24e6b965739e1fa390175 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:39.832886 22098 raft_consensus.cc:729] T be06870d950d419eb09d8557a45830dd P 1373fae1f2c24e6b965739e1fa390175 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1373fae1f2c24e6b965739e1fa390175, State: Initialized, Role: FOLLOWER
I20260928 06:33:39.832963 22101 tablet_bootstrap.cc:492] T be06870d950d419eb09d8557a45830dd P 36197995c3dd478eba3cf60405bc61d8: Bootstrap starting.
I20260928 06:33:39.832994 22098 consensus_queue.cc:260] T be06870d950d419eb09d8557a45830dd P 1373fae1f2c24e6b965739e1fa390175 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36197995c3dd478eba3cf60405bc61d8" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 33451 } } peers { permanent_uuid: "1373fae1f2c24e6b965739e1fa390175" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 36359 } } peers { permanent_uuid: "f6e3549d46d2493c900331c107cea0bd" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 36527 } }
I20260928 06:33:39.833225 22098 ts_tablet_manager.cc:1428] T be06870d950d419eb09d8557a45830dd P 1373fae1f2c24e6b965739e1fa390175: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:33:39.833319 21951 heartbeater.cc:503] Master 127.16.157.126:33073 was elected leader, sending a full tablet report...
I20260928 06:33:39.833976 22101 tablet_bootstrap.cc:654] T be06870d950d419eb09d8557a45830dd P 36197995c3dd478eba3cf60405bc61d8: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:39.834275 22101 log.cc:824] T be06870d950d419eb09d8557a45830dd P 36197995c3dd478eba3cf60405bc61d8: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:39.834908 22101 tablet_bootstrap.cc:492] T be06870d950d419eb09d8557a45830dd P 36197995c3dd478eba3cf60405bc61d8: No bootstrap required, opened a new log
I20260928 06:33:39.834986 22101 ts_tablet_manager.cc:1397] T be06870d950d419eb09d8557a45830dd P 36197995c3dd478eba3cf60405bc61d8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:33:39.836939 22101 raft_consensus.cc:348] T be06870d950d419eb09d8557a45830dd P 36197995c3dd478eba3cf60405bc61d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36197995c3dd478eba3cf60405bc61d8" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 33451 } } peers { permanent_uuid: "1373fae1f2c24e6b965739e1fa390175" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 36359 } } peers { permanent_uuid: "f6e3549d46d2493c900331c107cea0bd" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 36527 } }
I20260928 06:33:39.837086 22101 raft_consensus.cc:374] T be06870d950d419eb09d8557a45830dd P 36197995c3dd478eba3cf60405bc61d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:39.837132 22101 raft_consensus.cc:729] T be06870d950d419eb09d8557a45830dd P 36197995c3dd478eba3cf60405bc61d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36197995c3dd478eba3cf60405bc61d8, State: Initialized, Role: FOLLOWER
I20260928 06:33:39.837244 22101 consensus_queue.cc:260] T be06870d950d419eb09d8557a45830dd P 36197995c3dd478eba3cf60405bc61d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36197995c3dd478eba3cf60405bc61d8" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 33451 } } peers { permanent_uuid: "1373fae1f2c24e6b965739e1fa390175" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 36359 } } peers { permanent_uuid: "f6e3549d46d2493c900331c107cea0bd" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 36527 } }
I20260928 06:33:39.837491 21820 heartbeater.cc:503] Master 127.16.157.126:33073 was elected leader, sending a full tablet report...
I20260928 06:33:39.837711 22101 ts_tablet_manager.cc:1428] T be06870d950d419eb09d8557a45830dd P 36197995c3dd478eba3cf60405bc61d8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:33:39.877238 22103 raft_consensus.cc:482] T be06870d950d419eb09d8557a45830dd P 1373fae1f2c24e6b965739e1fa390175 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:33:39.877434 22103 raft_consensus.cc:504] T be06870d950d419eb09d8557a45830dd P 1373fae1f2c24e6b965739e1fa390175 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36197995c3dd478eba3cf60405bc61d8" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 33451 } } peers { permanent_uuid: "1373fae1f2c24e6b965739e1fa390175" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 36359 } } peers { permanent_uuid: "f6e3549d46d2493c900331c107cea0bd" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 36527 } }
I20260928 06:33:39.877873 22103 leader_election.cc:288] T be06870d950d419eb09d8557a45830dd P 1373fae1f2c24e6b965739e1fa390175 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36197995c3dd478eba3cf60405bc61d8 (127.16.157.65:33451), f6e3549d46d2493c900331c107cea0bd (127.16.157.67:36527)
I20260928 06:33:39.883307 21953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:39.878070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.65:33451 (local address 127.16.157.66:48771)
0928 06:33:39.878195 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:39.878207 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:39.878253 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:39.878465 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:39.878467 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:39.878470 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:39.882927 (+ 4457us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:33:39.882928 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:39.882936 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:33:39.882951 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:39.883163 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:39.883164 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:39.883165 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 06:33:39.883213 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 06:33:39.883243 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107}
I20260928 06:33:39.883303 22095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:39.878211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.66:48771 (local address 127.16.157.65:33451)
0928 06:33:39.878266 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:39.878268 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:39.878279 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:33:39.878311 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:39.878312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:39.878319 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:39.878391 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:39.883069 (+ 4678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:39.883072 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:39.883072 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:33:39.883077 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:39.883097 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:33:39.883102 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:39.883153 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:39.883212 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 06:33:39.883228 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 06:33:39.883319 22094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:39.878140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.66:37763 (local address 127.16.157.67:36527)
0928 06:33:39.878193 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:39.878195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:39.878456 (+ 261us) server_negotiation.cc:407] Connection header received
0928 06:33:39.878487 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:39.878488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:39.878494 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:39.878581 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:39.883023 (+ 4442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:39.883025 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:39.883026 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:33:39.883031 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:39.883052 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:33:39.883057 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:39.883111 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:39.883211 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 06:33:39.883245 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 06:33:39.883612 22106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:39.878070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.67:36527 (local address 127.16.157.66:37763)
0928 06:33:39.878380 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:39.878389 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:39.878434 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:39.878594 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:39.878595 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:39.878597 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:39.882884 (+ 4287us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:33:39.882885 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:39.882895 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:33:39.882913 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:39.883121 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:39.883122 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:39.883123 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 06:33:39.883522 (+ 399us) client_negotiation.cc:240] Negotiation successful
0928 06:33:39.883536 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":146,"thread_start_us":112,"threads_started":1}
I20260928 06:33:39.883705 21774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be06870d950d419eb09d8557a45830dd" candidate_uuid: "1373fae1f2c24e6b965739e1fa390175" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36197995c3dd478eba3cf60405bc61d8" is_pre_election: true
I20260928 06:33:39.883863 21774 raft_consensus.cc:2454] T be06870d950d419eb09d8557a45830dd P 36197995c3dd478eba3cf60405bc61d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1373fae1f2c24e6b965739e1fa390175 in term 0.
I20260928 06:33:39.884126 22036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be06870d950d419eb09d8557a45830dd" candidate_uuid: "1373fae1f2c24e6b965739e1fa390175" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6e3549d46d2493c900331c107cea0bd" is_pre_election: true
I20260928 06:33:39.884279 22036 raft_consensus.cc:2454] T be06870d950d419eb09d8557a45830dd P f6e3549d46d2493c900331c107cea0bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1373fae1f2c24e6b965739e1fa390175 in term 0.
I20260928 06:33:39.884560 21840 leader_election.cc:302] T be06870d950d419eb09d8557a45830dd P 1373fae1f2c24e6b965739e1fa390175 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1373fae1f2c24e6b965739e1fa390175, 36197995c3dd478eba3cf60405bc61d8; no voters:
I20260928 06:33:39.884763 22103 raft_consensus.cc:2781] T be06870d950d419eb09d8557a45830dd P 1373fae1f2c24e6b965739e1fa390175 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:33:39.884809 22103 raft_consensus.cc:482] T be06870d950d419eb09d8557a45830dd P 1373fae1f2c24e6b965739e1fa390175 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:33:39.884824 22103 raft_consensus.cc:3037] T be06870d950d419eb09d8557a45830dd P 1373fae1f2c24e6b965739e1fa390175 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:39.885638 22103 raft_consensus.cc:504] T be06870d950d419eb09d8557a45830dd P 1373fae1f2c24e6b965739e1fa390175 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36197995c3dd478eba3cf60405bc61d8" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 33451 } } peers { permanent_uuid: "1373fae1f2c24e6b965739e1fa390175" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 36359 } } peers { permanent_uuid: "f6e3549d46d2493c900331c107cea0bd" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 36527 } }
I20260928 06:33:39.885792 22103 leader_election.cc:288] T be06870d950d419eb09d8557a45830dd P 1373fae1f2c24e6b965739e1fa390175 [CANDIDATE]: Term 1 election: Requested vote from peers 36197995c3dd478eba3cf60405bc61d8 (127.16.157.65:33451), f6e3549d46d2493c900331c107cea0bd (127.16.157.67:36527)
I20260928 06:33:39.885965 22036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be06870d950d419eb09d8557a45830dd" candidate_uuid: "1373fae1f2c24e6b965739e1fa390175" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6e3549d46d2493c900331c107cea0bd"
I20260928 06:33:39.885958 21774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be06870d950d419eb09d8557a45830dd" candidate_uuid: "1373fae1f2c24e6b965739e1fa390175" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36197995c3dd478eba3cf60405bc61d8"
I20260928 06:33:39.886036 21774 raft_consensus.cc:3037] T be06870d950d419eb09d8557a45830dd P 36197995c3dd478eba3cf60405bc61d8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:39.886044 22036 raft_consensus.cc:3037] T be06870d950d419eb09d8557a45830dd P f6e3549d46d2493c900331c107cea0bd [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:39.886920 22036 raft_consensus.cc:2454] T be06870d950d419eb09d8557a45830dd P f6e3549d46d2493c900331c107cea0bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1373fae1f2c24e6b965739e1fa390175 in term 1.
I20260928 06:33:39.886921 21774 raft_consensus.cc:2454] T be06870d950d419eb09d8557a45830dd P 36197995c3dd478eba3cf60405bc61d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1373fae1f2c24e6b965739e1fa390175 in term 1.
I20260928 06:33:39.887101 21840 leader_election.cc:302] T be06870d950d419eb09d8557a45830dd P 1373fae1f2c24e6b965739e1fa390175 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1373fae1f2c24e6b965739e1fa390175, 36197995c3dd478eba3cf60405bc61d8; no voters:
I20260928 06:33:39.887194 22103 raft_consensus.cc:2781] T be06870d950d419eb09d8557a45830dd P 1373fae1f2c24e6b965739e1fa390175 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:33:39.887352 22103 raft_consensus.cc:686] T be06870d950d419eb09d8557a45830dd P 1373fae1f2c24e6b965739e1fa390175 [term 1 LEADER]: Becoming Leader. State: Replica: 1373fae1f2c24e6b965739e1fa390175, State: Running, Role: LEADER
I20260928 06:33:39.887457 22103 consensus_queue.cc:237] T be06870d950d419eb09d8557a45830dd P 1373fae1f2c24e6b965739e1fa390175 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36197995c3dd478eba3cf60405bc61d8" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 33451 } } peers { permanent_uuid: "1373fae1f2c24e6b965739e1fa390175" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 36359 } } peers { permanent_uuid: "f6e3549d46d2493c900331c107cea0bd" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 36527 } }
I20260928 06:33:39.888285 21632 catalog_manager.cc:5495] T be06870d950d419eb09d8557a45830dd P 1373fae1f2c24e6b965739e1fa390175 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1373fae1f2c24e6b965739e1fa390175 (127.16.157.66). New cstate: current_term: 1 leader_uuid: "1373fae1f2c24e6b965739e1fa390175" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36197995c3dd478eba3cf60405bc61d8" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 33451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1373fae1f2c24e6b965739e1fa390175" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 36359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6e3549d46d2493c900331c107cea0bd" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 36527 } health_report { overall_health: UNKNOWN } } }
I20260928 06:33:39.900460 22096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:39.899441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36754 (local address 127.16.157.66:36359)
0928 06:33:39.899509 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:39.899511 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:39.899527 (+ 16us) server_negotiation.cc:407] Connection header received
0928 06:33:39.899568 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:39.899569 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:39.899579 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:39.899638 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:39.900186 (+ 548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:39.900188 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:39.900188 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:33:39.900192 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:39.900208 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:33:39.900212 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:39.900264 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:39.900345 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 06:33:39.900358 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 06:33:39.902351 21774 raft_consensus.cc:1261] T be06870d950d419eb09d8557a45830dd P 36197995c3dd478eba3cf60405bc61d8 [term 1 FOLLOWER]: Refusing update from remote peer 1373fae1f2c24e6b965739e1fa390175: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:33:39.902351 22036 raft_consensus.cc:1261] T be06870d950d419eb09d8557a45830dd P f6e3549d46d2493c900331c107cea0bd [term 1 FOLLOWER]: Refusing update from remote peer 1373fae1f2c24e6b965739e1fa390175: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:33:39.902725 22107 consensus_queue.cc:1035] T be06870d950d419eb09d8557a45830dd P 1373fae1f2c24e6b965739e1fa390175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6e3549d46d2493c900331c107cea0bd" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 36527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:33:39.902858 22103 consensus_queue.cc:1035] T be06870d950d419eb09d8557a45830dd P 1373fae1f2c24e6b965739e1fa390175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36197995c3dd478eba3cf60405bc61d8" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 33451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:33:39.905217 22110 mvcc.cc:205] Tried to move back new op lower bound from 7334204292717809664 to 7334204292660322304. Current Snapshot: MvccSnapshot[applied={T|T < 7334204292717809664}]
I20260928 06:33:39.905666 22112 mvcc.cc:205] Tried to move back new op lower bound from 7334204292717809664 to 7334204292660322304. Current Snapshot: MvccSnapshot[applied={T|T < 7334204292717809664}]
I20260928 06:33:39.906629 22111 mvcc.cc:205] Tried to move back new op lower bound from 7334204292717809664 to 7334204292660322304. Current Snapshot: MvccSnapshot[applied={T|T < 7334204292717809664}]
I20260928 06:33:39.907701 21632 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:33:39.907804 21632 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:33:39.908538 21632 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c271cf65405b44e69bcef1b1bd71eb0b: Sending DeleteTablet for 3 replicas of tablet be06870d950d419eb09d8557a45830dd
I20260928 06:33:39.908972 21885 tablet_service.cc:1514] Processing DeleteTablet for tablet be06870d950d419eb09d8557a45830dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:33:39 UTC) from {username='slave'} at 127.0.0.1:36740
I20260928 06:33:39.908964 22016 tablet_service.cc:1514] Processing DeleteTablet for tablet be06870d950d419eb09d8557a45830dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:33:39 UTC) from {username='slave'} at 127.0.0.1:46092
I20260928 06:33:39.909292 21754 tablet_service.cc:1514] Processing DeleteTablet for tablet be06870d950d419eb09d8557a45830dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:33:39 UTC) from {username='slave'} at 127.0.0.1:53198
I20260928 06:33:39.909632 22119 tablet_replica.cc:330] T be06870d950d419eb09d8557a45830dd P f6e3549d46d2493c900331c107cea0bd: stopping tablet replica
I20260928 06:33:39.909727 22119 raft_consensus.cc:2229] T be06870d950d419eb09d8557a45830dd P f6e3549d46d2493c900331c107cea0bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:33:39.909816 22119 raft_consensus.cc:2258] T be06870d950d419eb09d8557a45830dd P f6e3549d46d2493c900331c107cea0bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:33:39.909884 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 21692
I20260928 06:33:39.910005 22122 tablet_replica.cc:330] T be06870d950d419eb09d8557a45830dd P 36197995c3dd478eba3cf60405bc61d8: stopping tablet replica
I20260928 06:33:39.910094 22122 raft_consensus.cc:2229] T be06870d950d419eb09d8557a45830dd P 36197995c3dd478eba3cf60405bc61d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:33:39.910192 22122 raft_consensus.cc:2258] T be06870d950d419eb09d8557a45830dd P 36197995c3dd478eba3cf60405bc61d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:33:39.910744 22119 ts_tablet_manager.cc:1905] T be06870d950d419eb09d8557a45830dd P f6e3549d46d2493c900331c107cea0bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:33:39.912446 22119 ts_tablet_manager.cc:1918] T be06870d950d419eb09d8557a45830dd P f6e3549d46d2493c900331c107cea0bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:33:39.912508 22119 log.cc:1196] T be06870d950d419eb09d8557a45830dd P f6e3549d46d2493c900331c107cea0bd: Deleting WAL directory at /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577196946045-17013-0/minicluster-data/ts-2/wal/wals/be06870d950d419eb09d8557a45830dd
I20260928 06:33:39.912758 22119 ts_tablet_manager.cc:1939] T be06870d950d419eb09d8557a45830dd P f6e3549d46d2493c900331c107cea0bd: Deleting consensus metadata
I20260928 06:33:39.912977 22121 tablet_replica.cc:330] T be06870d950d419eb09d8557a45830dd P 1373fae1f2c24e6b965739e1fa390175: stopping tablet replica
I20260928 06:33:39.913079 22121 raft_consensus.cc:2229] T be06870d950d419eb09d8557a45830dd P 1373fae1f2c24e6b965739e1fa390175 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:33:39.913169 22121 raft_consensus.cc:2258] T be06870d950d419eb09d8557a45830dd P 1373fae1f2c24e6b965739e1fa390175 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:33:39.913297 21617 catalog_manager.cc:4841] TS f6e3549d46d2493c900331c107cea0bd (127.16.157.67:36527): tablet be06870d950d419eb09d8557a45830dd (table test-table [id=0a213699963f41c28efdd03a0a732e62]) successfully deleted
I20260928 06:33:39.913812 22121 ts_tablet_manager.cc:1905] T be06870d950d419eb09d8557a45830dd P 1373fae1f2c24e6b965739e1fa390175: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:33:39.915387 22121 ts_tablet_manager.cc:1918] T be06870d950d419eb09d8557a45830dd P 1373fae1f2c24e6b965739e1fa390175: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:33:39.915459 22121 log.cc:1196] T be06870d950d419eb09d8557a45830dd P 1373fae1f2c24e6b965739e1fa390175: Deleting WAL directory at /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577196946045-17013-0/minicluster-data/ts-1/wal/wals/be06870d950d419eb09d8557a45830dd
I20260928 06:33:39.915728 22121 ts_tablet_manager.cc:1939] T be06870d950d419eb09d8557a45830dd P 1373fae1f2c24e6b965739e1fa390175: Deleting consensus metadata
I20260928 06:33:39.916126 21618 catalog_manager.cc:4841] TS 1373fae1f2c24e6b965739e1fa390175 (127.16.157.66:36359): tablet be06870d950d419eb09d8557a45830dd (table test-table [id=0a213699963f41c28efdd03a0a732e62]) successfully deleted
I20260928 06:33:39.916981 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 21823
W20260928 06:33:39.919046 21618 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.157.65:33451 (error 108)
I20260928 06:33:39.919519 22091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:39.919374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.65:33451 (local address 127.0.0.1:53204)
0928 06:33:39.919423 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:39.919479 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.65:33451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260928 06:33:39.919664 21618 catalog_manager.cc:4584] TS 36197995c3dd478eba3cf60405bc61d8 (127.16.157.65:33451): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet be06870d950d419eb09d8557a45830dd: Network error: Client connection negotiation failed: client connection to 127.16.157.65:33451: connect: Connection refused (error 111)
I20260928 06:33:39.923406 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 21954
I20260928 06:33:39.928727 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 21601
2026-09-28T06:33:39Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (658 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 06:33:39.950012 17013 mini_oidc.cc:152] Starting JWKS server
I20260928 06:33:39.951143 17013 webserver.cc:460] Webserver started at https://127.0.0.1:35089/ using document root <none> and password file <none>
I20260928 06:33:39.951265 17013 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 06:33:39.951464 17013 webserver.cc:460] Webserver started at http://127.0.0.1:44857/ using document root <none> and password file <none>
2026-09-28T06:33:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:33:39Z Disabled control of system clock
I20260928 06:33:39.960477 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577196946045-17013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577196946045-17013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577196946045-17013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:42315
--webserver_interface=127.16.157.126
--webserver_port=0
--builtin_ntp_servers=127.16.157.84:43467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.157.126:42315
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35089/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577196946045-17013-0/testchainca.cert with env {}
W20260928 06:33:40.044571 22129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:40.044759 22129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:40.044780 22129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:40.046000 22129 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:33:40.046046 22129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:40.046060 22129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:40.046092 22129 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:33:40.046123 22129 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 06:33:40.046139 22129 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 06:33:40.046164 22129 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35089/jwks.json
I20260928 06:33:40.047735 22129 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:43467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577196946045-17013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577196946045-17013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.157.126:42315
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:42315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35089/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.157.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577196946045-17013-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577196946045-17013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:40.047981 22129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:40.048206 22129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:40.053198 22134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:40.053200 22135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:40.053205 22137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:40.053401 22129 server_base.cc:1034] running on GCE node
I20260928 06:33:40.053705 22129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:40.054044 22129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:40.055207 22129 hybrid_clock.cc:648] HybridClock initialized: now 1790577220055189 us; error 44 us; skew 500 ppm
I20260928 06:33:40.056453 22129 webserver.cc:460] Webserver started at http://127.16.157.126:33903/ using document root <none> and password file <none>
I20260928 06:33:40.056717 22129 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:40.056792 22129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:40.056937 22129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:40.058024 22129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577196946045-17013-0/minicluster-data/master-0/data/instance:
uuid: "34899887d43044d48af3d287420a7425"
format_stamp: "Formatted at 2026-09-28 06:33:40 on dist-test-slave-rrj5"
server_key: "b19c5975e4b21db9bd62d35fe682aa80"
server_key_iv: "fb7ba9d072a71bffa2829fda655ac20f"
server_key_version: "encryptionkey@0"
I20260928 06:33:40.058383 22129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577196946045-17013-0/minicluster-data/master-0/wal/instance:
uuid: "34899887d43044d48af3d287420a7425"
format_stamp: "Formatted at 2026-09-28 06:33:40 on dist-test-slave-rrj5"
server_key: "b19c5975e4b21db9bd62d35fe682aa80"
server_key_iv: "fb7ba9d072a71bffa2829fda655ac20f"
server_key_version: "encryptionkey@0"
I20260928 06:33:40.059727 22129 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:33:40.060539 22143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:40.060729 22129 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:40.060842 22129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577196946045-17013-0/minicluster-data/master-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577196946045-17013-0/minicluster-data/master-0/wal
uuid: "34899887d43044d48af3d287420a7425"
format_stamp: "Formatted at 2026-09-28 06:33:40 on dist-test-slave-rrj5"
server_key: "b19c5975e4b21db9bd62d35fe682aa80"
server_key_iv: "fb7ba9d072a71bffa2829fda655ac20f"
server_key_version: "encryptionkey@0"
I20260928 06:33:40.060925 22129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577196946045-17013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577196946045-17013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577196946045-17013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:40.093786 22129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:40.094125 22129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:40.094285 22129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:40.098474 22129 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.126:42315
I20260928 06:33:40.098526 22195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.126:42315 every 8 connection(s)
I20260928 06:33:40.098886 22129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
I20260928 06:33:40.099977 22197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:40.103160 22197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34899887d43044d48af3d287420a7425: Bootstrap starting.
I20260928 06:33:40.103863 22197 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34899887d43044d48af3d287420a7425: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:40.104214 22197 log.cc:824] T 00000000000000000000000000000000 P 34899887d43044d48af3d287420a7425: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:40.104971 22197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34899887d43044d48af3d287420a7425: No bootstrap required, opened a new log
I20260928 06:33:40.107519 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 22129
I20260928 06:33:40.107633 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577196946045-17013-0/minicluster-data/master-0/wal/instance
I20260928 06:33:40.107796 17013 external_mini_cluster.cc:1428] Setting key 9bb6735fce9837939748f975cca880aa
I20260928 06:33:40.107662 22197 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34899887d43044d48af3d287420a7425 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34899887d43044d48af3d287420a7425" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 42315 } }
I20260928 06:33:40.107841 22197 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34899887d43044d48af3d287420a7425 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:40.107885 22197 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34899887d43044d48af3d287420a7425 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34899887d43044d48af3d287420a7425, State: Initialized, Role: FOLLOWER
I20260928 06:33:40.108027 22197 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34899887d43044d48af3d287420a7425 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34899887d43044d48af3d287420a7425" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 42315 } }
I20260928 06:33:40.108105 22197 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34899887d43044d48af3d287420a7425 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:33:40.108137 22197 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34899887d43044d48af3d287420a7425 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:33:40.108184 22197 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34899887d43044d48af3d287420a7425 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:40.108989 22197 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34899887d43044d48af3d287420a7425 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34899887d43044d48af3d287420a7425" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 42315 } }
I20260928 06:33:40.109119 22197 leader_election.cc:302] T 00000000000000000000000000000000 P 34899887d43044d48af3d287420a7425 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34899887d43044d48af3d287420a7425; no voters:
I20260928 06:33:40.109284 22197 leader_election.cc:288] T 00000000000000000000000000000000 P 34899887d43044d48af3d287420a7425 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:33:40.109310 22202 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34899887d43044d48af3d287420a7425 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:33:40.109419 22202 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34899887d43044d48af3d287420a7425 [term 1 LEADER]: Becoming Leader. State: Replica: 34899887d43044d48af3d287420a7425, State: Running, Role: LEADER
I20260928 06:33:40.109548 22197 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34899887d43044d48af3d287420a7425 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:33:40.109491 22202 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34899887d43044d48af3d287420a7425 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34899887d43044d48af3d287420a7425" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 42315 } }
I20260928 06:33:40.110387 22202 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34899887d43044d48af3d287420a7425 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34899887d43044d48af3d287420a7425" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34899887d43044d48af3d287420a7425" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 42315 } } }
I20260928 06:33:40.110450 22202 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34899887d43044d48af3d287420a7425 [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:40.110551 22202 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34899887d43044d48af3d287420a7425 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34899887d43044d48af3d287420a7425. Latest consensus state: current_term: 1 leader_uuid: "34899887d43044d48af3d287420a7425" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34899887d43044d48af3d287420a7425" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 42315 } } }
I20260928 06:33:40.110606 22202 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34899887d43044d48af3d287420a7425 [sys.catalog]: This master's current role is: LEADER
W20260928 06:33:40.111405 22217 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 34899887d43044d48af3d287420a7425: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:33:40.111461 22217 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:33:40.111534 22211 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:33:40.111721 22211 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:33:40.113327 22211 catalog_manager.cc:1347] Generated new cluster ID: eafe2e75eae8471a8909d06486508891
I20260928 06:33:40.113377 22211 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:33:40.115953 22201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:40.108422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44876 (local address 127.16.157.126:42315)
0928 06:33:40.108655 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:40.108663 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:40.108687 (+ 24us) server_negotiation.cc:407] Connection header received
0928 06:33:40.108778 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:40.108784 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:40.108845 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:40.109318 (+ 473us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:40.109442 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:40.112234 (+ 2792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:40.113653 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:40.114245 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:40.114342 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:40.115212 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:40.115216 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:40.115216 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:33:40.115409 (+ 193us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:40.115447 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:33:40.115459 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:40.115509 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:40.115682 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 06:33:40.115751 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":119,"threads_started":1}
I20260928 06:33:40.118402 17013 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:33:40.149976 22211 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:33:40.150712 22211 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:33:40.157558 22211 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34899887d43044d48af3d287420a7425: Generated new TSK 0
I20260928 06:33:40.157763 22211 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 06:33:40.189508 22227 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 06:33:40.189774 22227 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260928 06:33:40.190212 22226 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:33:40.167761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:42315 (local address 127.0.0.1:44888)
0928 06:33:40.167961 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:40.167974 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:40.168055 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:40.168277 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:40.168278 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:40.168285 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0928 06:33:40.168362 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:40.168365 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:40.169317 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:40.169318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:40.170277 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:40.170280 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:40.170778 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:40.170779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:40.170849 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:40.170871 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 06:33:40.189789 (+ 18918us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35089/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 06:33:40.189899 (+ 110us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.157.126:42315: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35089/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":103,"threads_started":1}
I20260928 06:33:40.190506 22201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:40.167819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44888 (local address 127.16.157.126:42315)
0928 06:33:40.167876 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:40.167878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:40.168051 (+ 173us) server_negotiation.cc:407] Connection header received
0928 06:33:40.168156 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:40.168157 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:40.168173 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:40.168261 (+ 88us) server_negotiation.cc:226] Negotiated authn=JWT
0928 06:33:40.168442 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:40.169219 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:40.170371 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:40.170679 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:40.170756 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:40.171004 (+ 248us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 06:33:40.189631 (+ 18627us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35089/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 06:33:40.190447 (+ 816us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44888: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35089/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":28}
W20260928 06:33:40.190560 22201 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44888: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35089/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.16.157.126:42315: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35089/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 06:33:40.191093 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 22129
2026-09-28T06:33:40Z chronyd exiting
I20260928 06:33:40.227301 17013 test_util.cc:175] -----------------------------------------------
I20260928 06:33:40.227401 17013 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577196946045-17013-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (278 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 06:33:40.228149 17013 mini_oidc.cc:152] Starting JWKS server
I20260928 06:33:40.228858 17013 webserver.cc:460] Webserver started at https://127.0.0.1:33251/ using document root <none> and password file <none>
I20260928 06:33:40.228929 17013 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 06:33:40.229161 17013 webserver.cc:460] Webserver started at http://127.0.0.1:37673/ using document root <none> and password file <none>
2026-09-28T06:33:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:33:40Z Disabled control of system clock
I20260928 06:33:40.237370 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577196946045-17013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577196946045-17013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577196946045-17013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:43469
--webserver_interface=127.16.157.126
--webserver_port=0
--builtin_ntp_servers=127.16.157.84:38293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.157.126:43469
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33251/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577196946045-17013-0/testchainca.cert with env {}
W20260928 06:33:40.318313 22233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:40.318517 22233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:40.318539 22233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:40.319711 22233 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:33:40.319782 22233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:40.319793 22233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:40.319806 22233 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:33:40.319818 22233 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 06:33:40.319868 22233 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 06:33:40.319886 22233 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33251/jwks.json
I20260928 06:33:40.321427 22233 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:38293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577196946045-17013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577196946045-17013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.157.126:43469
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:43469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33251/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.157.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577196946045-17013-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577196946045-17013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:40.321692 22233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:40.321926 22233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:40.326817 22241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:40.326825 22239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:40.326829 22238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:40.327374 22233 server_base.cc:1034] running on GCE node
I20260928 06:33:40.327649 22233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:40.327950 22233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:40.329125 22233 hybrid_clock.cc:648] HybridClock initialized: now 1790577220329106 us; error 35 us; skew 500 ppm
I20260928 06:33:40.330554 22233 webserver.cc:460] Webserver started at http://127.16.157.126:46107/ using document root <none> and password file <none>
I20260928 06:33:40.330819 22233 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:40.330888 22233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:40.331051 22233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:40.332542 22233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577196946045-17013-0/minicluster-data/master-0/data/instance:
uuid: "20f63cc271ff4e22935fd8995778232e"
format_stamp: "Formatted at 2026-09-28 06:33:40 on dist-test-slave-rrj5"
server_key: "ed77dd76f97ed0af186172466a8e051f"
server_key_iv: "ab527032746e596a34cd6bb24f306e59"
server_key_version: "encryptionkey@0"
I20260928 06:33:40.333011 22233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577196946045-17013-0/minicluster-data/master-0/wal/instance:
uuid: "20f63cc271ff4e22935fd8995778232e"
format_stamp: "Formatted at 2026-09-28 06:33:40 on dist-test-slave-rrj5"
server_key: "ed77dd76f97ed0af186172466a8e051f"
server_key_iv: "ab527032746e596a34cd6bb24f306e59"
server_key_version: "encryptionkey@0"
I20260928 06:33:40.334687 22233 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:33:40.335534 22247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:40.335839 22233 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:40.335935 22233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577196946045-17013-0/minicluster-data/master-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577196946045-17013-0/minicluster-data/master-0/wal
uuid: "20f63cc271ff4e22935fd8995778232e"
format_stamp: "Formatted at 2026-09-28 06:33:40 on dist-test-slave-rrj5"
server_key: "ed77dd76f97ed0af186172466a8e051f"
server_key_iv: "ab527032746e596a34cd6bb24f306e59"
server_key_version: "encryptionkey@0"
I20260928 06:33:40.336014 22233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577196946045-17013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577196946045-17013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577196946045-17013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:40.348181 22233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:40.348471 22233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:40.348635 22233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:40.352798 22233 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.126:43469
I20260928 06:33:40.352859 22299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.126:43469 every 8 connection(s)
I20260928 06:33:40.353246 22233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
I20260928 06:33:40.354009 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 22233
I20260928 06:33:40.354036 22300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:40.354126 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577196946045-17013-0/minicluster-data/master-0/wal/instance
I20260928 06:33:40.354313 17013 external_mini_cluster.cc:1428] Setting key c75df75cd354fa85324b586c40a42f35
I20260928 06:33:40.358242 22300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20f63cc271ff4e22935fd8995778232e: Bootstrap starting.
I20260928 06:33:40.358916 22300 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20f63cc271ff4e22935fd8995778232e: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:40.359225 22300 log.cc:824] T 00000000000000000000000000000000 P 20f63cc271ff4e22935fd8995778232e: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:40.359983 22300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20f63cc271ff4e22935fd8995778232e: No bootstrap required, opened a new log
I20260928 06:33:40.360194 22303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:40.354850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42004 (local address 127.16.157.126:43469)
0928 06:33:40.355078 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:40.355085 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:40.355109 (+ 24us) server_negotiation.cc:407] Connection header received
0928 06:33:40.355953 (+ 844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:40.355960 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:40.356018 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:40.356161 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:40.356325 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:40.357175 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:40.358052 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:40.358539 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:40.358671 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:40.359422 (+ 751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:40.359427 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:40.359428 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:33:40.359632 (+ 204us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:40.359676 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:33:40.359700 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:40.359760 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:40.359836 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 06:33:40.359911 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":94,"threads_started":1}
I20260928 06:33:40.361085 17013 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:33:40.362376 22300 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20f63cc271ff4e22935fd8995778232e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20f63cc271ff4e22935fd8995778232e" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 43469 } }
I20260928 06:33:40.362538 22300 raft_consensus.cc:374] T 00000000000000000000000000000000 P 20f63cc271ff4e22935fd8995778232e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:40.362571 22300 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20f63cc271ff4e22935fd8995778232e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20f63cc271ff4e22935fd8995778232e, State: Initialized, Role: FOLLOWER
I20260928 06:33:40.362682 22300 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20f63cc271ff4e22935fd8995778232e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20f63cc271ff4e22935fd8995778232e" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 43469 } }
I20260928 06:33:40.362751 22300 raft_consensus.cc:388] T 00000000000000000000000000000000 P 20f63cc271ff4e22935fd8995778232e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:33:40.362776 22300 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20f63cc271ff4e22935fd8995778232e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:33:40.362811 22300 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20f63cc271ff4e22935fd8995778232e [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:40.363432 22300 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20f63cc271ff4e22935fd8995778232e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20f63cc271ff4e22935fd8995778232e" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 43469 } }
I20260928 06:33:40.363534 22300 leader_election.cc:302] T 00000000000000000000000000000000 P 20f63cc271ff4e22935fd8995778232e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20f63cc271ff4e22935fd8995778232e; no voters:
I20260928 06:33:40.363686 22300 leader_election.cc:288] T 00000000000000000000000000000000 P 20f63cc271ff4e22935fd8995778232e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:33:40.363780 22305 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20f63cc271ff4e22935fd8995778232e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:33:40.363972 22300 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20f63cc271ff4e22935fd8995778232e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:33:40.363960 22305 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20f63cc271ff4e22935fd8995778232e [term 1 LEADER]: Becoming Leader. State: Replica: 20f63cc271ff4e22935fd8995778232e, State: Running, Role: LEADER
I20260928 06:33:40.364118 22305 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20f63cc271ff4e22935fd8995778232e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20f63cc271ff4e22935fd8995778232e" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 43469 } }
I20260928 06:33:40.364497 22305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20f63cc271ff4e22935fd8995778232e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20f63cc271ff4e22935fd8995778232e. Latest consensus state: current_term: 1 leader_uuid: "20f63cc271ff4e22935fd8995778232e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20f63cc271ff4e22935fd8995778232e" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 43469 } } }
I20260928 06:33:40.364576 22305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20f63cc271ff4e22935fd8995778232e [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:40.364727 22306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20f63cc271ff4e22935fd8995778232e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "20f63cc271ff4e22935fd8995778232e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20f63cc271ff4e22935fd8995778232e" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 43469 } } }
I20260928 06:33:40.364789 22306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20f63cc271ff4e22935fd8995778232e [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:40.365444 22314 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:33:40.366029 22314 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:33:40.367655 22314 catalog_manager.cc:1347] Generated new cluster ID: 8ac19d022662465486ccb326475c1276
I20260928 06:33:40.367709 22314 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:33:40.382772 22314 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:33:40.383422 22314 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:33:40.390050 22314 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 20f63cc271ff4e22935fd8995778232e: Generated new TSK 0
I20260928 06:33:40.390249 22314 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 06:33:40.419853 22330 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 06:33:40.420011 22330 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260928 06:33:40.420423 22329 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:33:40.397767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:43469 (local address 127.0.0.1:42006)
0928 06:33:40.397966 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:40.397977 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:40.398043 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:40.398209 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:40.398209 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:40.398212 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0928 06:33:40.398277 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:40.398280 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:40.399137 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:40.399138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:40.400068 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:40.400071 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:40.400554 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:40.400555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:40.400627 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:40.400633 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 06:33:40.420235 (+ 19602us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33251/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 06:33:40.420328 (+ 93us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.157.126:43469: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33251/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":95,"threads_started":1}
I20260928 06:33:40.420720 22303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:40.397848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42006 (local address 127.16.157.126:43469)
0928 06:33:40.397903 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:40.397905 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:40.398045 (+ 140us) server_negotiation.cc:407] Connection header received
0928 06:33:40.398108 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:40.398109 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:40.398123 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:40.398206 (+ 83us) server_negotiation.cc:226] Negotiated authn=JWT
0928 06:33:40.398344 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:40.399039 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:40.400162 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:40.400463 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:40.400536 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:40.400735 (+ 199us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 06:33:40.420029 (+ 19294us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33251/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 06:33:40.420647 (+ 618us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42006: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33251/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":31}
W20260928 06:33:40.420768 22303 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42006: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33251/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260928 06:33:40.420897 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 22233
2026-09-28T06:33:40Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (231 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 06:33:40.459739 17013 mini_oidc.cc:152] Starting JWKS server
I20260928 06:33:40.460476 17013 webserver.cc:460] Webserver started at https://127.0.0.1:44427/ using document root <none> and password file <none>
I20260928 06:33:40.460536 17013 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 06:33:40.460724 17013 webserver.cc:460] Webserver started at http://127.0.0.1:43629/ using document root <none> and password file <none>
2026-09-28T06:33:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:33:40Z Disabled control of system clock
I20260928 06:33:40.469009 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577196946045-17013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577196946045-17013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577196946045-17013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:36551
--webserver_interface=127.16.157.126
--webserver_port=0
--builtin_ntp_servers=127.16.157.84:37513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.157.126:36551
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44427/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 06:33:40.548740 22337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:40.548946 22337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:40.548966 22337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:40.550158 22337 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:33:40.550199 22337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:40.550212 22337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:40.550225 22337 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:33:40.550236 22337 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 06:33:40.550287 22337 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 06:33:40.550303 22337 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44427/jwks.json
I20260928 06:33:40.551743 22337 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:37513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577196946045-17013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577196946045-17013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.157.126:36551
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:36551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44427/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577196946045-17013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:40.551977 22337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:40.552250 22337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:40.556975 22345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:40.556958 22343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:40.556958 22342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:40.557117 22337 server_base.cc:1034] running on GCE node
I20260928 06:33:40.557490 22337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:40.557829 22337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:40.558985 22337 hybrid_clock.cc:648] HybridClock initialized: now 1790577220558975 us; error 28 us; skew 500 ppm
I20260928 06:33:40.560285 22337 webserver.cc:460] Webserver started at http://127.16.157.126:41643/ using document root <none> and password file <none>
I20260928 06:33:40.560535 22337 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:40.560604 22337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:40.560743 22337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:40.561875 22337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577196946045-17013-0/minicluster-data/master-0/data/instance:
uuid: "d29f3c8a3a664e368520eb858d5dd445"
format_stamp: "Formatted at 2026-09-28 06:33:40 on dist-test-slave-rrj5"
server_key: "b0efc7f77b1a2f8c72be4ac6d60eb2a2"
server_key_iv: "50dc21a397cf2b36c40fb77daba00abc"
server_key_version: "encryptionkey@0"
I20260928 06:33:40.562229 22337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577196946045-17013-0/minicluster-data/master-0/wal/instance:
uuid: "d29f3c8a3a664e368520eb858d5dd445"
format_stamp: "Formatted at 2026-09-28 06:33:40 on dist-test-slave-rrj5"
server_key: "b0efc7f77b1a2f8c72be4ac6d60eb2a2"
server_key_iv: "50dc21a397cf2b36c40fb77daba00abc"
server_key_version: "encryptionkey@0"
I20260928 06:33:40.563580 22337 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:33:40.564328 22351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:40.564561 22337 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:40.564656 22337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577196946045-17013-0/minicluster-data/master-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577196946045-17013-0/minicluster-data/master-0/wal
uuid: "d29f3c8a3a664e368520eb858d5dd445"
format_stamp: "Formatted at 2026-09-28 06:33:40 on dist-test-slave-rrj5"
server_key: "b0efc7f77b1a2f8c72be4ac6d60eb2a2"
server_key_iv: "50dc21a397cf2b36c40fb77daba00abc"
server_key_version: "encryptionkey@0"
I20260928 06:33:40.564731 22337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577196946045-17013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577196946045-17013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577196946045-17013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:40.588754 22337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:40.589145 22337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:40.589300 22337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:40.594424 22337 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.126:36551
I20260928 06:33:40.594449 22403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.126:36551 every 8 connection(s)
I20260928 06:33:40.594854 22337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
I20260928 06:33:40.595537 22404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:40.598310 22404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d29f3c8a3a664e368520eb858d5dd445: Bootstrap starting.
I20260928 06:33:40.598896 22404 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d29f3c8a3a664e368520eb858d5dd445: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:40.599176 22404 log.cc:824] T 00000000000000000000000000000000 P d29f3c8a3a664e368520eb858d5dd445: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:40.599926 22404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d29f3c8a3a664e368520eb858d5dd445: No bootstrap required, opened a new log
I20260928 06:33:40.601493 22404 raft_consensus.cc:348] T 00000000000000000000000000000000 P d29f3c8a3a664e368520eb858d5dd445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d29f3c8a3a664e368520eb858d5dd445" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 36551 } }
I20260928 06:33:40.601709 22404 raft_consensus.cc:374] T 00000000000000000000000000000000 P d29f3c8a3a664e368520eb858d5dd445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:40.601787 22404 raft_consensus.cc:729] T 00000000000000000000000000000000 P d29f3c8a3a664e368520eb858d5dd445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d29f3c8a3a664e368520eb858d5dd445, State: Initialized, Role: FOLLOWER
I20260928 06:33:40.601962 22404 consensus_queue.cc:260] T 00000000000000000000000000000000 P d29f3c8a3a664e368520eb858d5dd445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d29f3c8a3a664e368520eb858d5dd445" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 36551 } }
I20260928 06:33:40.602059 22404 raft_consensus.cc:388] T 00000000000000000000000000000000 P d29f3c8a3a664e368520eb858d5dd445 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:33:40.602101 22404 raft_consensus.cc:482] T 00000000000000000000000000000000 P d29f3c8a3a664e368520eb858d5dd445 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:33:40.602149 22404 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d29f3c8a3a664e368520eb858d5dd445 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:40.602656 22404 raft_consensus.cc:504] T 00000000000000000000000000000000 P d29f3c8a3a664e368520eb858d5dd445 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d29f3c8a3a664e368520eb858d5dd445" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 36551 } }
I20260928 06:33:40.602780 22404 leader_election.cc:302] T 00000000000000000000000000000000 P d29f3c8a3a664e368520eb858d5dd445 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d29f3c8a3a664e368520eb858d5dd445; no voters:
I20260928 06:33:40.602970 22404 leader_election.cc:288] T 00000000000000000000000000000000 P d29f3c8a3a664e368520eb858d5dd445 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:33:40.603016 22407 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d29f3c8a3a664e368520eb858d5dd445 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:33:40.603180 22407 raft_consensus.cc:686] T 00000000000000000000000000000000 P d29f3c8a3a664e368520eb858d5dd445 [term 1 LEADER]: Becoming Leader. State: Replica: d29f3c8a3a664e368520eb858d5dd445, State: Running, Role: LEADER
I20260928 06:33:40.603183 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 22337
I20260928 06:33:40.603245 22404 sys_catalog.cc:564] T 00000000000000000000000000000000 P d29f3c8a3a664e368520eb858d5dd445 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:33:40.603271 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577196946045-17013-0/minicluster-data/master-0/wal/instance
I20260928 06:33:40.603271 22407 consensus_queue.cc:237] T 00000000000000000000000000000000 P d29f3c8a3a664e368520eb858d5dd445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d29f3c8a3a664e368520eb858d5dd445" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 36551 } }
I20260928 06:33:40.603469 17013 external_mini_cluster.cc:1428] Setting key 9ac5eddd513005a6589460ecfc249888
I20260928 06:33:40.603643 22407 sys_catalog.cc:455] T 00000000000000000000000000000000 P d29f3c8a3a664e368520eb858d5dd445 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d29f3c8a3a664e368520eb858d5dd445" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d29f3c8a3a664e368520eb858d5dd445" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 36551 } } }
I20260928 06:33:40.603714 22407 sys_catalog.cc:458] T 00000000000000000000000000000000 P d29f3c8a3a664e368520eb858d5dd445 [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:40.603829 22407 sys_catalog.cc:455] T 00000000000000000000000000000000 P d29f3c8a3a664e368520eb858d5dd445 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d29f3c8a3a664e368520eb858d5dd445. Latest consensus state: current_term: 1 leader_uuid: "d29f3c8a3a664e368520eb858d5dd445" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d29f3c8a3a664e368520eb858d5dd445" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 36551 } } }
I20260928 06:33:40.603873 22407 sys_catalog.cc:458] T 00000000000000000000000000000000 P d29f3c8a3a664e368520eb858d5dd445 [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:40.604591 22413 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:33:40.605183 22413 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:33:40.610199 22418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:40.604082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55066 (local address 127.16.157.126:36551)
0928 06:33:40.604272 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:40.604283 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:40.604348 (+ 65us) server_negotiation.cc:407] Connection header received
0928 06:33:40.604518 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:40.604525 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:40.604600 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:40.605031 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:40.605141 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:40.606642 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:40.607768 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:40.608432 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:40.608565 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:40.609380 (+ 815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:40.609385 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:40.609385 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:33:40.609643 (+ 258us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:40.609691 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:33:40.609707 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:40.609779 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:40.609870 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 06:33:40.609954 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":86,"threads_started":1}
I20260928 06:33:40.611169 17013 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:33:40.614694 22413 catalog_manager.cc:1347] Generated new cluster ID: 778000ce23e0436ab61b928e4ad888f5
I20260928 06:33:40.614753 22413 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:33:40.619314 22413 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:33:40.619961 22413 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:33:40.624413 22413 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d29f3c8a3a664e368520eb858d5dd445: Generated new TSK 0
I20260928 06:33:40.624573 22413 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:33:40.635003 22418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:40.630277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55080 (local address 127.16.157.126:36551)
0928 06:33:40.630317 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:40.630319 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:40.630459 (+ 140us) server_negotiation.cc:407] Connection header received
0928 06:33:40.630509 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:40.630510 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:40.630523 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:40.630600 (+ 77us) server_negotiation.cc:226] Negotiated authn=JWT
0928 06:33:40.630755 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:40.631568 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:40.632699 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:40.633078 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:40.633154 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:40.633356 (+ 202us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 06:33:40.634784 (+ 1428us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44427/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 06:33:40.634928 (+ 144us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55080: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44427/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 06:33:40.635102 22433 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:33:40.630201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:36551 (local address 127.0.0.1:55080)
0928 06:33:40.630375 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:40.630386 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:40.630452 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:40.630603 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:40.630604 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:40.630606 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 06:33:40.630675 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:40.630677 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:40.631684 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:40.631685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:40.632597 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:40.632600 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:40.633170 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:40.633171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:40.633246 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:40.633251 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 06:33:40.634889 (+ 1638us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44427/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 06:33:40.634990 (+ 101us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.157.126:36551: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44427/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":92,"threads_started":1}
I20260928 06:33:40.635318 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 22337
E20260928 06:33:40.657406 22434 webserver.cc:533] Webserver: sslize failed:
E20260928 06:33:40.657502 22434 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-28T06:33:40Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (213 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T06:33:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:33:40Z Disabled control of system clock
I20260928 06:33:40.681445 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577196946045-17013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577196946045-17013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577196946045-17013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:43443
--webserver_interface=127.16.157.126
--webserver_port=0
--builtin_ntp_servers=127.16.157.84:41225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.157.126:43443
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577196946045-17013-0/testchainca.cert with env {}
W20260928 06:33:40.761790 22438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:40.761973 22438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:40.761996 22438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:40.763135 22438 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:33:40.763190 22438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:40.763202 22438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:40.763216 22438 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:33:40.763226 22438 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 06:33:40.763262 22438 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 06:33:40.763290 22438 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 06:33:40.764755 22438 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:41225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577196946045-17013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577196946045-17013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.157.126:43443
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:43443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.157.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577196946045-17013-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577196946045-17013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:40.765005 22438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:40.765226 22438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:40.770401 22444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:40.770426 22443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:40.770412 22446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:40.770663 22438 server_base.cc:1034] running on GCE node
I20260928 06:33:40.770886 22438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:40.771194 22438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:40.772359 22438 hybrid_clock.cc:648] HybridClock initialized: now 1790577220772350 us; error 30 us; skew 500 ppm
I20260928 06:33:40.773854 22438 webserver.cc:460] Webserver started at http://127.16.157.126:35351/ using document root <none> and password file <none>
I20260928 06:33:40.774112 22438 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:40.774156 22438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:40.774307 22438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:40.775362 22438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577196946045-17013-0/minicluster-data/master-0/data/instance:
uuid: "aac370181fe1464ba5ec81c08be22876"
format_stamp: "Formatted at 2026-09-28 06:33:40 on dist-test-slave-rrj5"
server_key: "80ccb447eec6c22ddad4b41bd7c1f20f"
server_key_iv: "f9469b1578653633aea2df3312b723e3"
server_key_version: "encryptionkey@0"
I20260928 06:33:40.775789 22438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577196946045-17013-0/minicluster-data/master-0/wal/instance:
uuid: "aac370181fe1464ba5ec81c08be22876"
format_stamp: "Formatted at 2026-09-28 06:33:40 on dist-test-slave-rrj5"
server_key: "80ccb447eec6c22ddad4b41bd7c1f20f"
server_key_iv: "f9469b1578653633aea2df3312b723e3"
server_key_version: "encryptionkey@0"
I20260928 06:33:40.777099 22438 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:33:40.777951 22452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:40.778208 22438 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:33:40.778312 22438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577196946045-17013-0/minicluster-data/master-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577196946045-17013-0/minicluster-data/master-0/wal
uuid: "aac370181fe1464ba5ec81c08be22876"
format_stamp: "Formatted at 2026-09-28 06:33:40 on dist-test-slave-rrj5"
server_key: "80ccb447eec6c22ddad4b41bd7c1f20f"
server_key_iv: "f9469b1578653633aea2df3312b723e3"
server_key_version: "encryptionkey@0"
I20260928 06:33:40.778403 22438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577196946045-17013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577196946045-17013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577196946045-17013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:40.796900 22438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:40.797200 22438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:40.797348 22438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:40.801082 22438 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.126:43443
I20260928 06:33:40.801151 22505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.126:43443 every 8 connection(s)
I20260928 06:33:40.801507 22438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
I20260928 06:33:40.802727 22506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:40.805857 22506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aac370181fe1464ba5ec81c08be22876: Bootstrap starting.
I20260928 06:33:40.805912 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 22438
I20260928 06:33:40.806032 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577196946045-17013-0/minicluster-data/master-0/wal/instance
I20260928 06:33:40.806243 17013 external_mini_cluster.cc:1428] Setting key aae69e6dc4ece807f0fe9e31fdebd825
I20260928 06:33:40.806576 22506 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aac370181fe1464ba5ec81c08be22876: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:40.806926 22506 log.cc:824] T 00000000000000000000000000000000 P aac370181fe1464ba5ec81c08be22876: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:40.807755 22506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aac370181fe1464ba5ec81c08be22876: No bootstrap required, opened a new log
I20260928 06:33:40.810165 22506 raft_consensus.cc:348] T 00000000000000000000000000000000 P aac370181fe1464ba5ec81c08be22876 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aac370181fe1464ba5ec81c08be22876" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 43443 } }
I20260928 06:33:40.810343 22506 raft_consensus.cc:374] T 00000000000000000000000000000000 P aac370181fe1464ba5ec81c08be22876 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:40.810388 22506 raft_consensus.cc:729] T 00000000000000000000000000000000 P aac370181fe1464ba5ec81c08be22876 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aac370181fe1464ba5ec81c08be22876, State: Initialized, Role: FOLLOWER
I20260928 06:33:40.810523 22506 consensus_queue.cc:260] T 00000000000000000000000000000000 P aac370181fe1464ba5ec81c08be22876 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aac370181fe1464ba5ec81c08be22876" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 43443 } }
I20260928 06:33:40.810629 22506 raft_consensus.cc:388] T 00000000000000000000000000000000 P aac370181fe1464ba5ec81c08be22876 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:33:40.810662 22506 raft_consensus.cc:482] T 00000000000000000000000000000000 P aac370181fe1464ba5ec81c08be22876 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:33:40.810710 22506 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aac370181fe1464ba5ec81c08be22876 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:40.811400 22506 raft_consensus.cc:504] T 00000000000000000000000000000000 P aac370181fe1464ba5ec81c08be22876 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aac370181fe1464ba5ec81c08be22876" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 43443 } }
I20260928 06:33:40.811522 22506 leader_election.cc:302] T 00000000000000000000000000000000 P aac370181fe1464ba5ec81c08be22876 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aac370181fe1464ba5ec81c08be22876; no voters:
I20260928 06:33:40.811690 22506 leader_election.cc:288] T 00000000000000000000000000000000 P aac370181fe1464ba5ec81c08be22876 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:33:40.811723 22511 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aac370181fe1464ba5ec81c08be22876 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:33:40.811829 22511 raft_consensus.cc:686] T 00000000000000000000000000000000 P aac370181fe1464ba5ec81c08be22876 [term 1 LEADER]: Becoming Leader. State: Replica: aac370181fe1464ba5ec81c08be22876, State: Running, Role: LEADER
I20260928 06:33:40.811914 22511 consensus_queue.cc:237] T 00000000000000000000000000000000 P aac370181fe1464ba5ec81c08be22876 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aac370181fe1464ba5ec81c08be22876" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 43443 } }
I20260928 06:33:40.811972 22506 sys_catalog.cc:564] T 00000000000000000000000000000000 P aac370181fe1464ba5ec81c08be22876 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:33:40.812268 22513 sys_catalog.cc:455] T 00000000000000000000000000000000 P aac370181fe1464ba5ec81c08be22876 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aac370181fe1464ba5ec81c08be22876" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aac370181fe1464ba5ec81c08be22876" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 43443 } } }
I20260928 06:33:40.812386 22513 sys_catalog.cc:458] T 00000000000000000000000000000000 P aac370181fe1464ba5ec81c08be22876 [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:40.812266 22512 sys_catalog.cc:455] T 00000000000000000000000000000000 P aac370181fe1464ba5ec81c08be22876 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aac370181fe1464ba5ec81c08be22876. Latest consensus state: current_term: 1 leader_uuid: "aac370181fe1464ba5ec81c08be22876" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aac370181fe1464ba5ec81c08be22876" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 43443 } } }
I20260928 06:33:40.812515 22512 sys_catalog.cc:458] T 00000000000000000000000000000000 P aac370181fe1464ba5ec81c08be22876 [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:40.813261 22517 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:33:40.813449 22509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:40.806767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44146 (local address 127.16.157.126:43443)
0928 06:33:40.807019 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:40.807029 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:40.807065 (+ 36us) server_negotiation.cc:407] Connection header received
0928 06:33:40.807615 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:40.807623 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:40.807687 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:40.807840 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:40.808046 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:40.808927 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:40.809797 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:40.810218 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:40.810502 (+ 284us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:40.811190 (+ 688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:40.811194 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:40.811194 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:33:40.811376 (+ 182us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:40.811408 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:33:40.811419 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:40.811503 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:40.811746 (+ 243us) server_negotiation.cc:299] Negotiation successful
0928 06:33:40.811838 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":96,"threads_started":1}
I20260928 06:33:40.813798 22517 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:33:40.814390 17013 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:33:40.818704 22517 catalog_manager.cc:1347] Generated new cluster ID: b47631a58ddf4edbac21babc37a42e77
I20260928 06:33:40.818756 22517 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:33:40.849339 22517 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:33:40.849961 22517 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:33:40.856307 22517 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aac370181fe1464ba5ec81c08be22876: Generated new TSK 0
I20260928 06:33:40.856499 22517 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:33:40.893956 22509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:40.871858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44154 (local address 127.16.157.126:43443)
0928 06:33:40.871901 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:40.871903 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:40.872040 (+ 137us) server_negotiation.cc:407] Connection header received
0928 06:33:40.872146 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:40.872147 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:40.872158 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:40.872201 (+ 43us) server_negotiation.cc:226] Negotiated authn=JWT
0928 06:33:40.872386 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:40.873040 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:40.874082 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:40.874357 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:40.874393 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:40.874664 (+ 271us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 06:33:40.893690 (+ 19026us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 06:33:40.893873 (+ 183us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44154: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 06:33:40.894030 22509 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44154: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260928 06:33:40.894119 22535 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:33:40.868259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:43443 (local address 127.0.0.1:44154)
0928 06:33:40.871973 (+ 3714us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:40.871984 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:40.872067 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:40.872259 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:40.872260 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:40.872261 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0928 06:33:40.872321 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:40.872322 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:40.873169 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:40.873170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:40.873981 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:40.873984 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:40.874494 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:40.874495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:40.874563 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:40.874568 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 06:33:40.893843 (+ 19275us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 06:33:40.893993 (+ 150us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.157.126:43443: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":3687,"thread_start_us":101,"threads_started":1}
I20260928 06:33:40.894403 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 22438
2026-09-28T06:33:40Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (239 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (90 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (91 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:33:41 dist-test-slave-rrj5 krb5kdc[22547](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:33:41 dist-test-slave-rrj5 krb5kdc[22547](info): set up 2 sockets
Sep 28 06:33:41 dist-test-slave-rrj5 krb5kdc[22547](info): commencing operation
krb5kdc: starting...
W20260928 06:33:42.175091 17013 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Sep 28 06:33:42 dist-test-slave-rrj5 krb5kdc[22547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:33:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:33:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.157.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab.
Entry for principal kudu/127.16.157.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab.
I20260928 06:33:42.310629 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:46791
--webserver_interface=127.16.157.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=kudu/127.16.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:44539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.157.126:46791
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:42.403448 22560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:42.403733 22560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:42.403765 22560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:42.405051 22560 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:33:42.405092 22560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:42.405108 22560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:42.405143 22560 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:33:42.405169 22560 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:33:42.406769 22560 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:44539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.157.126:46791
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=kudu/127.16.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:46791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:42.407015 22560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:42.407321 22560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:42.412592 22566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:42.412588 22568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:42.412842 22560 server_base.cc:1034] running on GCE node
W20260928 06:33:42.412873 22565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:42.413183 22560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:42.413511 22560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:42.414705 22560 hybrid_clock.cc:648] HybridClock initialized: now 1790577222414675 us; error 41 us; skew 500 ppm
Sep 28 06:33:42 dist-test-slave-rrj5 krb5kdc[22547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577222, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:42.417896 22560 init.cc:377] Logged in from keytab as kudu/127.16.157.126@KRBTEST.COM (short username kudu)
I20260928 06:33:42.418458 22560 webserver.cc:460] Webserver started at http://127.16.157.126:35845/ using document root <none> and password file <none>
I20260928 06:33:42.418764 22560 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:42.418828 22560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:42.418956 22560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:42.420099 22560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/minicluster-data/master-0/data/instance:
uuid: "5fc2df23d0a0485aaa173f9b2a9a3b24"
format_stamp: "Formatted at 2026-09-28 06:33:42 on dist-test-slave-rrj5"
server_key: "cbad22c8346c9e23ea7a6881d2e96624"
server_key_iv: "c3c9aa6ccde18413d9a5d22958c7d9a0"
server_key_version: "encryptionkey@0"
I20260928 06:33:42.420473 22560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/minicluster-data/master-0/wal/instance:
uuid: "5fc2df23d0a0485aaa173f9b2a9a3b24"
format_stamp: "Formatted at 2026-09-28 06:33:42 on dist-test-slave-rrj5"
server_key: "cbad22c8346c9e23ea7a6881d2e96624"
server_key_iv: "c3c9aa6ccde18413d9a5d22958c7d9a0"
server_key_version: "encryptionkey@0"
I20260928 06:33:42.422169 22560 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:33:42.423089 22575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:42.423251 22560 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:42.423359 22560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/minicluster-data/master-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/minicluster-data/master-0/wal
uuid: "5fc2df23d0a0485aaa173f9b2a9a3b24"
format_stamp: "Formatted at 2026-09-28 06:33:42 on dist-test-slave-rrj5"
server_key: "cbad22c8346c9e23ea7a6881d2e96624"
server_key_iv: "c3c9aa6ccde18413d9a5d22958c7d9a0"
server_key_version: "encryptionkey@0"
I20260928 06:33:42.423429 22560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:42.444833 22560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:42.447402 22560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:42.447583 22560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:42.452687 22560 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.126:46791
I20260928 06:33:42.452742 22628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.126:46791 every 8 connection(s)
I20260928 06:33:42.453181 22560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
I20260928 06:33:42.454221 22629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:42.455374 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 22560
I20260928 06:33:42.455482 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/minicluster-data/master-0/wal/instance
I20260928 06:33:42.455711 17013 external_mini_cluster.cc:1428] Setting key e18708e21e46b409c05042abf8c34c0e
I20260928 06:33:42.457394 22629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fc2df23d0a0485aaa173f9b2a9a3b24: Bootstrap starting.
I20260928 06:33:42.458117 22629 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5fc2df23d0a0485aaa173f9b2a9a3b24: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:42.458455 22629 log.cc:824] T 00000000000000000000000000000000 P 5fc2df23d0a0485aaa173f9b2a9a3b24: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:42.460640 22629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fc2df23d0a0485aaa173f9b2a9a3b24: No bootstrap required, opened a new log
I20260928 06:33:42.462970 22629 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5fc2df23d0a0485aaa173f9b2a9a3b24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc2df23d0a0485aaa173f9b2a9a3b24" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 46791 } }
I20260928 06:33:42.463155 22629 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5fc2df23d0a0485aaa173f9b2a9a3b24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:42.463204 22629 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5fc2df23d0a0485aaa173f9b2a9a3b24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fc2df23d0a0485aaa173f9b2a9a3b24, State: Initialized, Role: FOLLOWER
I20260928 06:33:42.463346 22629 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5fc2df23d0a0485aaa173f9b2a9a3b24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc2df23d0a0485aaa173f9b2a9a3b24" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 46791 } }
I20260928 06:33:42.463446 22629 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5fc2df23d0a0485aaa173f9b2a9a3b24 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:33:42.463487 22629 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5fc2df23d0a0485aaa173f9b2a9a3b24 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:33:42.463542 22629 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5fc2df23d0a0485aaa173f9b2a9a3b24 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:42.464090 22629 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5fc2df23d0a0485aaa173f9b2a9a3b24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc2df23d0a0485aaa173f9b2a9a3b24" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 46791 } }
I20260928 06:33:42.464210 22629 leader_election.cc:302] T 00000000000000000000000000000000 P 5fc2df23d0a0485aaa173f9b2a9a3b24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fc2df23d0a0485aaa173f9b2a9a3b24; no voters:
I20260928 06:33:42.464383 22629 leader_election.cc:288] T 00000000000000000000000000000000 P 5fc2df23d0a0485aaa173f9b2a9a3b24 [CANDIDATE]: Term 1 election: Requested vote from peers
Sep 28 06:33:42 dist-test-slave-rrj5 krb5kdc[22547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:33:42.464401 22634 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5fc2df23d0a0485aaa173f9b2a9a3b24 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:33:42.464640 22634 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5fc2df23d0a0485aaa173f9b2a9a3b24 [term 1 LEADER]: Becoming Leader. State: Replica: 5fc2df23d0a0485aaa173f9b2a9a3b24, State: Running, Role: LEADER
I20260928 06:33:42.464790 22634 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5fc2df23d0a0485aaa173f9b2a9a3b24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc2df23d0a0485aaa173f9b2a9a3b24" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 46791 } }
I20260928 06:33:42.465142 22634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fc2df23d0a0485aaa173f9b2a9a3b24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5fc2df23d0a0485aaa173f9b2a9a3b24. Latest consensus state: current_term: 1 leader_uuid: "5fc2df23d0a0485aaa173f9b2a9a3b24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc2df23d0a0485aaa173f9b2a9a3b24" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 46791 } } }
I20260928 06:33:42.465219 22634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fc2df23d0a0485aaa173f9b2a9a3b24 [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:42.465314 22635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fc2df23d0a0485aaa173f9b2a9a3b24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5fc2df23d0a0485aaa173f9b2a9a3b24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc2df23d0a0485aaa173f9b2a9a3b24" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 46791 } } }
I20260928 06:33:42.464651 22629 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5fc2df23d0a0485aaa173f9b2a9a3b24 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:33:42.465445 22635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fc2df23d0a0485aaa173f9b2a9a3b24 [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:42.465512 22638 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:33:42.466064 22638 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:33:42.470685 22638 catalog_manager.cc:1347] Generated new cluster ID: 94abe5baa1234c6c982c692cd33dd80a
I20260928 06:33:42.470754 22638 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:33:42.471068 22632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:42.456286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44034 (local address 127.16.157.126:46791)
0928 06:33:42.456461 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:42.456470 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:42.456502 (+ 32us) server_negotiation.cc:407] Connection header received
0928 06:33:42.457379 (+ 877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:42.457390 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:42.457464 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:42.457712 (+ 248us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:42.458713 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:42.460000 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:42.461194 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:42.461846 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:42.461965 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:42.465072 (+ 3107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:42.465076 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:42.465076 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:42.465089 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:42.468711 (+ 3622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:42.469147 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:42.469151 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:42.469151 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:42.469299 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:42.469561 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:42.469562 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:42.469563 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:42.469967 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:42.470153 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:42.470733 (+ 580us) server_negotiation.cc:299] Negotiation successful
0928 06:33:42.470874 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":72,"threads_started":1}
I20260928 06:33:42.485070 22638 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:33:42.486047 22638 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:33:42.491547 22638 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5fc2df23d0a0485aaa173f9b2a9a3b24: Generated new TSK 0
I20260928 06:33:42.491756 22638 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.157.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab.
Entry for principal kudu/127.16.157.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab.
I20260928 06:33:42.532151 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.65:0
--local_ip_for_outbound_sockets=127.16.157.65
--webserver_interface=127.16.157.65
--webserver_port=0
--tserver_master_addrs=127.16.157.126:46791
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab
--principal=kudu/127.16.157.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:44539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:42.647665 22655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:42.647899 22655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:42.647939 22655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:42.649101 22655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:42.649166 22655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:42.649214 22655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.65
I20260928 06:33:42.650799 22655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:44539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab
--principal=kudu/127.16.157.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.157.65
--webserver_port=0
--tserver_master_addrs=127.16.157.126:46791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.65
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:42.651064 22655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:42.651322 22655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:42.656242 22660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:42.656282 22661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:42.656556 22663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:42.657629 22655 server_base.cc:1034] running on GCE node
I20260928 06:33:42.657832 22655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:42.658071 22655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:42.659235 22655 hybrid_clock.cc:648] HybridClock initialized: now 1790577222659215 us; error 36 us; skew 500 ppm
Sep 28 06:33:42 dist-test-slave-rrj5 krb5kdc[22547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577222, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:42.662613 22655 init.cc:377] Logged in from keytab as kudu/127.16.157.65@KRBTEST.COM (short username kudu)
I20260928 06:33:42.663115 22655 webserver.cc:460] Webserver started at http://127.16.157.65:38577/ using document root <none> and password file <none>
I20260928 06:33:42.663348 22655 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:42.663410 22655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:42.663546 22655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:42.665016 22655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/minicluster-data/ts-0/data/instance:
uuid: "cc21246d31994f7c816b4c482b4cb0a1"
format_stamp: "Formatted at 2026-09-28 06:33:42 on dist-test-slave-rrj5"
server_key: "4772e134c49373292b2b309e4e862804"
server_key_iv: "0ab6acb4ca4617dbefab86793a2205f2"
server_key_version: "encryptionkey@0"
I20260928 06:33:42.665521 22655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/minicluster-data/ts-0/wal/instance:
uuid: "cc21246d31994f7c816b4c482b4cb0a1"
format_stamp: "Formatted at 2026-09-28 06:33:42 on dist-test-slave-rrj5"
server_key: "4772e134c49373292b2b309e4e862804"
server_key_iv: "0ab6acb4ca4617dbefab86793a2205f2"
server_key_version: "encryptionkey@0"
I20260928 06:33:42.667407 22655 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260928 06:33:42.668316 22670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:42.668488 22655 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:42.668592 22655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/minicluster-data/ts-0/wal
uuid: "cc21246d31994f7c816b4c482b4cb0a1"
format_stamp: "Formatted at 2026-09-28 06:33:42 on dist-test-slave-rrj5"
server_key: "4772e134c49373292b2b309e4e862804"
server_key_iv: "0ab6acb4ca4617dbefab86793a2205f2"
server_key_version: "encryptionkey@0"
I20260928 06:33:42.668675 22655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:42.702610 22655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:42.705124 22655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:42.705296 22655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:42.705597 22655 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:33:42.705976 22655 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:33:42.706024 22655 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:42.706054 22655 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:33:42.706070 22655 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:42.713414 22655 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.65:37563
I20260928 06:33:42.713466 22783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.65:37563 every 8 connection(s)
I20260928 06:33:42.713835 22655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
I20260928 06:33:42.718216 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 22655
I20260928 06:33:42.718333 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/minicluster-data/ts-0/wal/instance
I20260928 06:33:42.718534 17013 external_mini_cluster.cc:1428] Setting key 6d58cb1eeeb9590301011ab464ac022e
Sep 28 06:33:42 dist-test-slave-rrj5 krb5kdc[22547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577222, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.65@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:33:42.725246 22632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:42.714560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:58867 (local address 127.16.157.126:46791)
0928 06:33:42.714601 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:42.714603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:42.714817 (+ 214us) server_negotiation.cc:407] Connection header received
0928 06:33:42.716018 (+ 1201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:42.716019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:42.716030 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:42.716119 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:42.716859 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:42.717592 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:42.719050 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:42.719378 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:42.719471 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:42.721878 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:42.721880 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:42.721880 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:42.721888 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:42.723632 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:42.724191 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:42.724193 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:42.724193 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:42.724253 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:42.724563 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:42.724564 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:42.724564 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:42.724684 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:42.724758 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:42.725053 (+ 295us) server_negotiation.cc:299] Negotiation successful
0928 06:33:42.725157 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 06:33:42.725533 22786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:42.714529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:46791 (local address 127.16.157.65:58867)
0928 06:33:42.714700 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:42.714743 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:42.715862 (+ 1119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:42.716186 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:42.716189 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:42.716522 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:42.716774 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:42.716780 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:42.717672 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:42.717673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:42.718956 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:42.718959 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:42.719480 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:42.719481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:42.719628 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:42.720171 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:42.720188 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:42.721778 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:42.723754 (+ 1976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:42.723761 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:42.723764 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:42.724090 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:42.724324 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:42.724324 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:42.724324 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:42.724500 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:42.724762 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:42.724769 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:42.724907 (+ 138us) client_negotiation.cc:769] Sending connection context
0928 06:33:42.725010 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 06:33:42.725193 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":94,"threads_started":1}
I20260928 06:33:42.726171 22784 heartbeater.cc:347] Connected to a master server at 127.16.157.126:46791
I20260928 06:33:42.726262 22784 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:42.726444 22784 heartbeater.cc:511] Master 127.16.157.126:46791 requested a full tablet report, sending...
I20260928 06:33:42.727046 22592 ts_manager.cc:194] Registered new tserver with Master: cc21246d31994f7c816b4c482b4cb0a1 (127.16.157.65:37563)
WARNING: no policy specified for kudu/127.16.157.66@KRBTEST.COM; defaulting to no policy
I20260928 06:33:42.727910 22592 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.157.65@KRBTEST.COM'} at 127.16.157.65:58867
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab.
Entry for principal kudu/127.16.157.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab.
I20260928 06:33:42.748262 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.66:0
--local_ip_for_outbound_sockets=127.16.157.66
--webserver_interface=127.16.157.66
--webserver_port=0
--tserver_master_addrs=127.16.157.126:46791
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab
--principal=kudu/127.16.157.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:44539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:42.858669 22789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:42.858937 22789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:42.858973 22789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:42.860669 22789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:42.860718 22789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:42.860759 22789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.66
I20260928 06:33:42.862911 22789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:44539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab
--principal=kudu/127.16.157.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.157.66
--webserver_port=0
--tserver_master_addrs=127.16.157.126:46791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.66
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:42.863158 22789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:42.863412 22789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:42.868844 22797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:42.868957 22794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:42.868840 22795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:42.869626 22789 server_base.cc:1034] running on GCE node
I20260928 06:33:42.869858 22789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:42.870167 22789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:42.871326 22789 hybrid_clock.cc:648] HybridClock initialized: now 1790577222871298 us; error 48 us; skew 500 ppm
Sep 28 06:33:42 dist-test-slave-rrj5 krb5kdc[22547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577222, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:42.876818 22789 init.cc:377] Logged in from keytab as kudu/127.16.157.66@KRBTEST.COM (short username kudu)
I20260928 06:33:42.877408 22789 webserver.cc:460] Webserver started at http://127.16.157.66:34601/ using document root <none> and password file <none>
I20260928 06:33:42.877746 22789 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:42.877823 22789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:42.877979 22789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:42.879474 22789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/minicluster-data/ts-1/data/instance:
uuid: "6e87ed6d00e5444e8e8c872031cc7422"
format_stamp: "Formatted at 2026-09-28 06:33:42 on dist-test-slave-rrj5"
server_key: "7246ee52d371573603253e7a1a5bf72d"
server_key_iv: "69f8506075d1f67afae8cbefaebf9280"
server_key_version: "encryptionkey@0"
I20260928 06:33:42.879918 22789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/minicluster-data/ts-1/wal/instance:
uuid: "6e87ed6d00e5444e8e8c872031cc7422"
format_stamp: "Formatted at 2026-09-28 06:33:42 on dist-test-slave-rrj5"
server_key: "7246ee52d371573603253e7a1a5bf72d"
server_key_iv: "69f8506075d1f67afae8cbefaebf9280"
server_key_version: "encryptionkey@0"
I20260928 06:33:42.881845 22789 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260928 06:33:42.882856 22804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:42.883158 22789 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:42.883265 22789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/minicluster-data/ts-1/wal
uuid: "6e87ed6d00e5444e8e8c872031cc7422"
format_stamp: "Formatted at 2026-09-28 06:33:42 on dist-test-slave-rrj5"
server_key: "7246ee52d371573603253e7a1a5bf72d"
server_key_iv: "69f8506075d1f67afae8cbefaebf9280"
server_key_version: "encryptionkey@0"
I20260928 06:33:42.883352 22789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:42.895056 22789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:42.897897 22789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:42.898064 22789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:42.898340 22789 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:33:42.898722 22789 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:33:42.898779 22789 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:42.898815 22789 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:33:42.898849 22789 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:42.906934 22789 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.66:43973
I20260928 06:33:42.906998 22917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.66:43973 every 8 connection(s)
I20260928 06:33:42.907342 22789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
I20260928 06:33:42.913411 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 22789
I20260928 06:33:42.913522 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/minicluster-data/ts-1/wal/instance
I20260928 06:33:42.913789 17013 external_mini_cluster.cc:1428] Setting key 586cc478f95b7d1c290f14503071dd07
Sep 28 06:33:42 dist-test-slave-rrj5 krb5kdc[22547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577222, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.66@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:33:42.921279 22632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:42.908145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.66:36415 (local address 127.16.157.126:46791)
0928 06:33:42.908223 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:42.908225 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:42.908495 (+ 270us) server_negotiation.cc:407] Connection header received
0928 06:33:42.909923 (+ 1428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:42.909925 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:42.909936 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:42.910003 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:42.910889 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:42.911685 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:42.913172 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:42.913530 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:42.913654 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:42.918281 (+ 4627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:42.918282 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:42.918282 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:42.918288 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:42.919740 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:42.920229 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:42.920231 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:42.920231 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:42.920292 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:42.920574 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:42.920575 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:42.920576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:42.920707 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:42.920749 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:42.921099 (+ 350us) server_negotiation.cc:299] Negotiation successful
0928 06:33:42.921187 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 06:33:42.921849 22920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:42.908113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:46791 (local address 127.16.157.66:36415)
0928 06:33:42.908342 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:42.908390 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:42.909803 (+ 1413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:42.910085 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:42.910090 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:42.910493 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:42.910784 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:42.910791 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:42.911795 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:42.911796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:42.913077 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:42.913080 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:42.914018 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:42.914019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:42.914176 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:42.914787 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:42.914797 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:42.918193 (+ 3396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:42.919832 (+ 1639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:42.919838 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:42.919840 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:42.920133 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:42.920362 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:42.920362 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:42.920363 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:42.920487 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:42.920794 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:42.920800 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:42.920949 (+ 149us) client_negotiation.cc:769] Sending connection context
0928 06:33:42.921407 (+ 458us) client_negotiation.cc:240] Negotiation successful
0928 06:33:42.921618 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":124,"threads_started":1}
I20260928 06:33:42.922314 22918 heartbeater.cc:347] Connected to a master server at 127.16.157.126:46791
I20260928 06:33:42.922448 22918 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:42.922664 22918 heartbeater.cc:511] Master 127.16.157.126:46791 requested a full tablet report, sending...
I20260928 06:33:42.923146 22592 ts_manager.cc:194] Registered new tserver with Master: 6e87ed6d00e5444e8e8c872031cc7422 (127.16.157.66:43973)
I20260928 06:33:42.923528 22592 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.157.66@KRBTEST.COM'} at 127.16.157.66:36415
WARNING: no policy specified for kudu/127.16.157.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab.
Entry for principal kudu/127.16.157.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab.
I20260928 06:33:42.945093 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.67:0
--local_ip_for_outbound_sockets=127.16.157.67
--webserver_interface=127.16.157.67
--webserver_port=0
--tserver_master_addrs=127.16.157.126:46791
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab
--principal=kudu/127.16.157.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:44539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:43.051872 22923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:43.052196 22923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:43.052241 22923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:43.054730 22923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:43.054803 22923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:43.054857 22923 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.67
I20260928 06:33:43.057718 22923 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:44539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab
--principal=kudu/127.16.157.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.157.67
--webserver_port=0
--tserver_master_addrs=127.16.157.126:46791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.67
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:43.057972 22923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:43.058216 22923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:43.063788 22931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:43.063818 22929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:43.063818 22928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:43.063911 22923 server_base.cc:1034] running on GCE node
I20260928 06:33:43.064235 22923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:43.064528 22923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:43.065702 22923 hybrid_clock.cc:648] HybridClock initialized: now 1790577223065687 us; error 43 us; skew 500 ppm
Sep 28 06:33:43 dist-test-slave-rrj5 krb5kdc[22547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577223, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:43.068706 22923 init.cc:377] Logged in from keytab as kudu/127.16.157.67@KRBTEST.COM (short username kudu)
I20260928 06:33:43.069271 22923 webserver.cc:460] Webserver started at http://127.16.157.67:40999/ using document root <none> and password file <none>
I20260928 06:33:43.069540 22923 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:43.069634 22923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:43.069785 22923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:43.071219 22923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/minicluster-data/ts-2/data/instance:
uuid: "8e4d3e12e6344e079f4fc4b3451549f6"
format_stamp: "Formatted at 2026-09-28 06:33:43 on dist-test-slave-rrj5"
server_key: "62351686f225cde5439a847bd0236172"
server_key_iv: "087ba57d6b5c5547f89b1b8079165674"
server_key_version: "encryptionkey@0"
I20260928 06:33:43.071664 22923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/minicluster-data/ts-2/wal/instance:
uuid: "8e4d3e12e6344e079f4fc4b3451549f6"
format_stamp: "Formatted at 2026-09-28 06:33:43 on dist-test-slave-rrj5"
server_key: "62351686f225cde5439a847bd0236172"
server_key_iv: "087ba57d6b5c5547f89b1b8079165674"
server_key_version: "encryptionkey@0"
I20260928 06:33:43.073459 22923 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260928 06:33:43.074412 22938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:43.074628 22923 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:43.074733 22923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/minicluster-data/ts-2/wal
uuid: "8e4d3e12e6344e079f4fc4b3451549f6"
format_stamp: "Formatted at 2026-09-28 06:33:43 on dist-test-slave-rrj5"
server_key: "62351686f225cde5439a847bd0236172"
server_key_iv: "087ba57d6b5c5547f89b1b8079165674"
server_key_version: "encryptionkey@0"
I20260928 06:33:43.074819 22923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:43.085470 22923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:43.087942 22923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:43.088099 22923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:43.088359 22923 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:33:43.088728 22923 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:33:43.088779 22923 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:43.088835 22923 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:33:43.088862 22923 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:43.096382 22923 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.67:46803
I20260928 06:33:43.096493 23052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.67:46803 every 8 connection(s)
I20260928 06:33:43.096791 22923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
I20260928 06:33:43.101063 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 22923
I20260928 06:33:43.101145 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/minicluster-data/ts-2/wal/instance
I20260928 06:33:43.101301 17013 external_mini_cluster.cc:1428] Setting key 481f3cacd80fe7cf69b0ae51fa094b58
Sep 28 06:33:43 dist-test-slave-rrj5 krb5kdc[22547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577223, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.67@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:33:43.106750 22632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:43.097632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:36515 (local address 127.16.157.126:46791)
0928 06:33:43.097689 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:43.097692 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:43.097865 (+ 173us) server_negotiation.cc:407] Connection header received
0928 06:33:43.098803 (+ 938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:43.098804 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:43.098817 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:43.098869 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:43.099459 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:43.099963 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:43.100984 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:43.101273 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:43.101318 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:43.103862 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:43.103863 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:43.103863 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:43.103870 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:43.105314 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:43.105818 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:43.105819 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:43.105819 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:43.105871 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:43.106146 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:43.106146 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:43.106146 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:43.106240 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:43.106284 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:43.106632 (+ 348us) server_negotiation.cc:299] Negotiation successful
0928 06:33:43.106691 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 06:33:43.107051 23055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:43.097583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:46791 (local address 127.16.157.67:36515)
0928 06:33:43.097740 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:43.097785 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:43.098683 (+ 898us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:43.098930 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:43.098934 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:43.099175 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:43.099347 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:43.099351 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:43.100058 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:43.100059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:43.100856 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:43.100858 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:43.101411 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:43.101412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:43.101597 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:43.102119 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:43.102129 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:43.103773 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:43.105390 (+ 1617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:43.105396 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:43.105398 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:43.105747 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:43.105937 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:43.105938 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:43.105938 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:43.106077 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:43.106301 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:43.106306 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:43.106466 (+ 160us) client_negotiation.cc:769] Sending connection context
0928 06:33:43.106603 (+ 137us) client_negotiation.cc:240] Negotiation successful
0928 06:33:43.106780 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":83,"threads_started":1}
I20260928 06:33:43.107559 23053 heartbeater.cc:347] Connected to a master server at 127.16.157.126:46791
I20260928 06:33:43.107671 23053 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:43.107857 23053 heartbeater.cc:511] Master 127.16.157.126:46791 requested a full tablet report, sending...
I20260928 06:33:43.108254 22592 ts_manager.cc:194] Registered new tserver with Master: 8e4d3e12e6344e079f4fc4b3451549f6 (127.16.157.67:46803)
I20260928 06:33:43.108693 22592 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.157.67@KRBTEST.COM'} at 127.16.157.67:36515
I20260928 06:33:43.112964 17013 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/krb5kdc/test-admin.keytab.
Sep 28 06:33:43 dist-test-slave-rrj5 krb5kdc[22547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:43.130847 17013 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 06:33:43.132540 23060 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:33:43.131687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39310 (local address 127.16.157.65:37563)
0928 06:33:43.131858 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:43.131862 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:43.131872 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:33:43.131894 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:43.131896 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:43.131909 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:43.132033 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:43.132455 (+ 422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39310: BlockingRecv error: recv got EOF from 127.0.0.1:39310 (error 108)
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":82,"threads_started":1}
W20260928 06:33:43.132652 17013 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/krb5kdc/krb5cc' of type FILE
Sep 28 06:33:43 dist-test-slave-rrj5 krb5kdc[22547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:43.133901 17013 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 06:33:43 dist-test-slave-rrj5 krb5kdc[22547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:43.134981 17013 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 06:33:43 dist-test-slave-rrj5 krb5kdc[22547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.157.65@KRBTEST.COM
I20260928 06:33:43.143868 23060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:43.135598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39322 (local address 127.16.157.65:37563)
0928 06:33:43.135632 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:43.135634 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:43.135680 (+ 46us) server_negotiation.cc:407] Connection header received
0928 06:33:43.135727 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:43.135728 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:43.135740 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:43.135802 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:43.136187 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:43.137427 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:43.138193 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:43.138743 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:43.138818 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:43.141163 (+ 2345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:43.141170 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:43.141170 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:43.141176 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:43.142715 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:43.143139 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:43.143141 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:43.143141 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:43.143190 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:43.143399 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:43.143399 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:43.143400 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:43.143528 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:43.143630 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:43.143752 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 06:33:43.143808 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
W20260928 06:33:43.145835 23060 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:33:43.145254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39326 (local address 127.16.157.65:37563)
0928 06:33:43.145291 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:43.145293 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:43.145376 (+ 83us) server_negotiation.cc:407] Connection header received
0928 06:33:43.145452 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:43.145453 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:43.145463 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:43.145511 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:43.145794 (+ 283us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39326: BlockingRecv error: recv got EOF from 127.0.0.1:39326 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 06:33:43.145973 17013 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/krb5kdc/krb5cc)
Sep 28 06:33:43 dist-test-slave-rrj5 krb5kdc[22547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:43.146960 17013 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 06:33:43 dist-test-slave-rrj5 krb5kdc[22547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:43.147818 17013 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 06:33:43 dist-test-slave-rrj5 krb5kdc[22547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.157.65@KRBTEST.COM
I20260928 06:33:43.154996 23060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:43.148324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39342 (local address 127.16.157.65:37563)
0928 06:33:43.148362 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:43.148364 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:43.148411 (+ 47us) server_negotiation.cc:407] Connection header received
0928 06:33:43.148485 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:43.148486 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:43.148493 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:43.148537 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:43.148875 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:43.149346 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:43.150130 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:43.150336 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:43.150408 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:43.152573 (+ 2165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:43.152574 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:43.152574 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:43.152581 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:43.153945 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:43.154285 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:43.154286 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:43.154286 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:43.154342 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:43.154542 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:43.154543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:43.154543 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:43.154639 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:43.154732 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:43.154830 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 06:33:43.154903 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 06:33:43.156447 23060 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:33:43.155932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39346 (local address 127.16.157.65:37563)
0928 06:33:43.155972 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:43.155974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:43.156001 (+ 27us) server_negotiation.cc:407] Connection header received
0928 06:33:43.156068 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:43.156069 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:43.156076 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:43.156119 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:43.156396 (+ 277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39346: BlockingRecv error: recv got EOF from 127.0.0.1:39346 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 06:33:43.156526 17013 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577196946045-17013-0/krb5kdc/krb5cc' of type FILE
Sep 28 06:33:43 dist-test-slave-rrj5 krb5kdc[22547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:43.157509 17013 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 06:33:43 dist-test-slave-rrj5 krb5kdc[22547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:43.158408 17013 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 06:33:43 dist-test-slave-rrj5 krb5kdc[22547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.157.65@KRBTEST.COM
I20260928 06:33:43.166399 23060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:43.158914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39352 (local address 127.16.157.65:37563)
0928 06:33:43.158950 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:43.158952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:43.159004 (+ 52us) server_negotiation.cc:407] Connection header received
0928 06:33:43.159064 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:43.159064 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:43.159073 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:43.159132 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:43.159482 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:43.159905 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:43.160565 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:43.160766 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:43.160838 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:43.163950 (+ 3112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:43.163951 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:43.163951 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:43.163957 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:43.165148 (+ 1191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:43.165589 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:43.165591 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:43.165591 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:43.165651 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:43.165869 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:43.165870 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:43.165870 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:43.165979 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:43.166066 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:43.166171 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 06:33:43.166250 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 06:33:43.167044 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 22655
I20260928 06:33:43.174216 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 22789
I20260928 06:33:43.179198 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 22923
I20260928 06:33:43.183430 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 22560
2026-09-28T06:33:43Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2109 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:33:43 dist-test-slave-rrj5 krb5kdc[23073](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:33:43 dist-test-slave-rrj5 krb5kdc[23073](info): set up 2 sockets
Sep 28 06:33:43 dist-test-slave-rrj5 krb5kdc[23073](info): commencing operation
krb5kdc: starting...
W20260928 06:33:44.302662 17013 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Sep 28 06:33:44 dist-test-slave-rrj5 krb5kdc[23073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:33:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:33:44Z Disabled control of system clock
WARNING: no policy specified for oryx/127.16.157.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.157.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.157.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/krb5kdc/oryx_127.16.157.126.keytab.
Entry for principal oryx/127.16.157.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/krb5kdc/oryx_127.16.157.126.keytab.
I20260928 06:33:44.424429 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:40365
--webserver_interface=127.16.157.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/krb5kdc/oryx_127.16.157.126.keytab
--principal=oryx/127.16.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:39253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.157.126:40365
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:44.503265 23086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:44.503446 23086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:44.503468 23086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:44.504570 23086 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:33:44.504611 23086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:44.504626 23086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:44.504638 23086 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:33:44.504649 23086 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:33:44.506176 23086 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:39253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.157.126:40365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/krb5kdc/oryx_127.16.157.126.keytab
--principal=oryx/127.16.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:40365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:44.506410 23086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:44.506646 23086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:44.511458 23092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:44.511497 23094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:44.511751 23086 server_base.cc:1034] running on GCE node
W20260928 06:33:44.511475 23091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:44.512025 23086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:44.512324 23086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:44.513509 23086 hybrid_clock.cc:648] HybridClock initialized: now 1790577224513484 us; error 41 us; skew 500 ppm
Sep 28 06:33:44 dist-test-slave-rrj5 krb5kdc[23073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577224, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:44.516139 23086 init.cc:377] Logged in from keytab as oryx/127.16.157.126@KRBTEST.COM (short username oryx)
I20260928 06:33:44.516592 23086 webserver.cc:460] Webserver started at http://127.16.157.126:41193/ using document root <none> and password file <none>
I20260928 06:33:44.516852 23086 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:44.516920 23086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:44.517067 23086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:44.518151 23086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/master-0/data/instance:
uuid: "3a9edcc4ed794902a9d229837466efd2"
format_stamp: "Formatted at 2026-09-28 06:33:44 on dist-test-slave-rrj5"
server_key: "395c44f62f5300585f92223b7c038518"
server_key_iv: "c07fc6beca5fcf3455912e3a8534c635"
server_key_version: "encryptionkey@0"
I20260928 06:33:44.518509 23086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/master-0/wal/instance:
uuid: "3a9edcc4ed794902a9d229837466efd2"
format_stamp: "Formatted at 2026-09-28 06:33:44 on dist-test-slave-rrj5"
server_key: "395c44f62f5300585f92223b7c038518"
server_key_iv: "c07fc6beca5fcf3455912e3a8534c635"
server_key_version: "encryptionkey@0"
I20260928 06:33:44.519852 23086 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:33:44.520633 23101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:44.520826 23086 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:44.520912 23086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/master-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/master-0/wal
uuid: "3a9edcc4ed794902a9d229837466efd2"
format_stamp: "Formatted at 2026-09-28 06:33:44 on dist-test-slave-rrj5"
server_key: "395c44f62f5300585f92223b7c038518"
server_key_iv: "c07fc6beca5fcf3455912e3a8534c635"
server_key_version: "encryptionkey@0"
I20260928 06:33:44.520983 23086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:44.545166 23086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:44.547276 23086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:44.547423 23086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:44.551523 23086 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.126:40365
I20260928 06:33:44.551590 23154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.126:40365 every 8 connection(s)
I20260928 06:33:44.551926 23086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
I20260928 06:33:44.553079 23155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:44.556198 23155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a9edcc4ed794902a9d229837466efd2: Bootstrap starting.
I20260928 06:33:44.556928 23155 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a9edcc4ed794902a9d229837466efd2: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:44.557246 23155 log.cc:824] T 00000000000000000000000000000000 P 3a9edcc4ed794902a9d229837466efd2: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:44.558709 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 23086
I20260928 06:33:44.558831 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/master-0/wal/instance
I20260928 06:33:44.559033 17013 external_mini_cluster.cc:1428] Setting key 13766edc05792a7275b808115629af32
Sep 28 06:33:44 dist-test-slave-rrj5 krb5kdc[23073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM
I20260928 06:33:44.569543 23159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:44.559562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48758 (local address 127.16.157.126:40365)
0928 06:33:44.559740 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:44.559748 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:44.559773 (+ 25us) server_negotiation.cc:407] Connection header received
0928 06:33:44.560279 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:44.560286 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:44.560335 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:44.560460 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:44.561216 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:44.562159 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:44.563245 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:44.563635 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:44.563710 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:44.566372 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:44.566376 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:44.566376 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:44.566386 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:44.568034 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:44.568440 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:44.568444 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:44.568444 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:44.568525 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:44.568733 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:44.568734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:44.568734 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:44.568974 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:44.569120 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:44.569229 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 06:33:44.569367 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":94,"threads_started":1}
I20260928 06:33:44.579166 23155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a9edcc4ed794902a9d229837466efd2: No bootstrap required, opened a new log
I20260928 06:33:44.581768 23155 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a9edcc4ed794902a9d229837466efd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a9edcc4ed794902a9d229837466efd2" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40365 } }
I20260928 06:33:44.581965 23155 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a9edcc4ed794902a9d229837466efd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:44.582008 23155 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a9edcc4ed794902a9d229837466efd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9edcc4ed794902a9d229837466efd2, State: Initialized, Role: FOLLOWER
I20260928 06:33:44.582137 23155 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a9edcc4ed794902a9d229837466efd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a9edcc4ed794902a9d229837466efd2" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40365 } }
I20260928 06:33:44.582211 23155 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a9edcc4ed794902a9d229837466efd2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:33:44.582242 23155 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a9edcc4ed794902a9d229837466efd2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:33:44.582293 23155 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a9edcc4ed794902a9d229837466efd2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:44.583062 23155 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a9edcc4ed794902a9d229837466efd2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a9edcc4ed794902a9d229837466efd2" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40365 } }
I20260928 06:33:44.583173 23155 leader_election.cc:302] T 00000000000000000000000000000000 P 3a9edcc4ed794902a9d229837466efd2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a9edcc4ed794902a9d229837466efd2; no voters:
I20260928 06:33:44.583341 23155 leader_election.cc:288] T 00000000000000000000000000000000 P 3a9edcc4ed794902a9d229837466efd2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:33:44.583413 23160 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a9edcc4ed794902a9d229837466efd2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:33:44.583611 23160 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a9edcc4ed794902a9d229837466efd2 [term 1 LEADER]: Becoming Leader. State: Replica: 3a9edcc4ed794902a9d229837466efd2, State: Running, Role: LEADER
I20260928 06:33:44.583619 23155 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a9edcc4ed794902a9d229837466efd2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:33:44.583734 23160 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a9edcc4ed794902a9d229837466efd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a9edcc4ed794902a9d229837466efd2" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40365 } }
I20260928 06:33:44.584213 23160 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a9edcc4ed794902a9d229837466efd2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a9edcc4ed794902a9d229837466efd2. Latest consensus state: current_term: 1 leader_uuid: "3a9edcc4ed794902a9d229837466efd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a9edcc4ed794902a9d229837466efd2" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40365 } } }
I20260928 06:33:44.584287 23160 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a9edcc4ed794902a9d229837466efd2 [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:44.584249 23161 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a9edcc4ed794902a9d229837466efd2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a9edcc4ed794902a9d229837466efd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a9edcc4ed794902a9d229837466efd2" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40365 } } }
I20260928 06:33:44.584342 23161 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a9edcc4ed794902a9d229837466efd2 [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:44.584497 23168 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:33:44.585171 23168 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:33:44.586787 23168 catalog_manager.cc:1347] Generated new cluster ID: fc73c8eec1414312b3fe0164efd101a8
I20260928 06:33:44.586848 23168 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:33:44.600620 23168 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:33:44.601150 23168 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:33:44.603330 23168 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a9edcc4ed794902a9d229837466efd2: Generated new TSK 0
I20260928 06:33:44.603508 23168 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.16.157.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.157.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.157.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/krb5kdc/oryx_127.16.157.65.keytab.
Entry for principal oryx/127.16.157.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/krb5kdc/oryx_127.16.157.65.keytab.
I20260928 06:33:44.631693 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.65:0
--local_ip_for_outbound_sockets=127.16.157.65
--webserver_interface=127.16.157.65
--webserver_port=0
--tserver_master_addrs=127.16.157.126:40365
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/krb5kdc/oryx_127.16.157.65.keytab
--principal=oryx/127.16.157.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:39253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:44.713238 23182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:44.713465 23182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:44.713507 23182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:44.714650 23182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:44.714727 23182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:44.714771 23182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.65
I20260928 06:33:44.716272 23182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:39253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/krb5kdc/oryx_127.16.157.65.keytab
--principal=oryx/127.16.157.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.157.65
--webserver_port=0
--tserver_master_addrs=127.16.157.126:40365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.65
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:44.716550 23182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:44.716753 23182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:44.721998 23188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:44.722112 23182 server_base.cc:1034] running on GCE node
W20260928 06:33:44.722011 23190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:44.722287 23187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:44.722512 23182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:44.722777 23182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:44.723930 23182 hybrid_clock.cc:648] HybridClock initialized: now 1790577224723912 us; error 42 us; skew 500 ppm
Sep 28 06:33:44 dist-test-slave-rrj5 krb5kdc[23073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577224, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.157.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:44.726593 23182 init.cc:377] Logged in from keytab as oryx/127.16.157.65@KRBTEST.COM (short username oryx)
I20260928 06:33:44.727084 23182 webserver.cc:460] Webserver started at http://127.16.157.65:35469/ using document root <none> and password file <none>
I20260928 06:33:44.727325 23182 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:44.727392 23182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:44.727531 23182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:44.728561 23182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-0/data/instance:
uuid: "d8d60b2a738a49288570af1432d7c119"
format_stamp: "Formatted at 2026-09-28 06:33:44 on dist-test-slave-rrj5"
server_key: "bb0a23ec650544e64ab1d00707f008cb"
server_key_iv: "60d321c371f770cf654fd6c051a0f00e"
server_key_version: "encryptionkey@0"
I20260928 06:33:44.728960 23182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-0/wal/instance:
uuid: "d8d60b2a738a49288570af1432d7c119"
format_stamp: "Formatted at 2026-09-28 06:33:44 on dist-test-slave-rrj5"
server_key: "bb0a23ec650544e64ab1d00707f008cb"
server_key_iv: "60d321c371f770cf654fd6c051a0f00e"
server_key_version: "encryptionkey@0"
I20260928 06:33:44.730387 23182 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:33:44.731156 23197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:44.731315 23182 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:33:44.731400 23182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-0/wal
uuid: "d8d60b2a738a49288570af1432d7c119"
format_stamp: "Formatted at 2026-09-28 06:33:44 on dist-test-slave-rrj5"
server_key: "bb0a23ec650544e64ab1d00707f008cb"
server_key_iv: "60d321c371f770cf654fd6c051a0f00e"
server_key_version: "encryptionkey@0"
I20260928 06:33:44.731492 23182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:44.737941 23182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:44.740108 23182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:44.740265 23182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:44.740504 23182 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:33:44.740846 23182 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:33:44.740906 23182 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:44.740950 23182 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:33:44.740976 23182 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:44.748092 23182 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.65:36871
I20260928 06:33:44.748131 23311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.65:36871 every 8 connection(s)
I20260928 06:33:44.748481 23182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
Sep 28 06:33:44 dist-test-slave-rrj5 krb5kdc[23073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577224, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.157.65@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM
I20260928 06:33:44.756482 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 23182
I20260928 06:33:44.756584 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-0/wal/instance
I20260928 06:33:44.756779 17013 external_mini_cluster.cc:1428] Setting key 912009c64f2f6ecc609bfa2d2dda22e1
I20260928 06:33:44.760160 23159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:44.749176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:37379 (local address 127.16.157.126:40365)
0928 06:33:44.749420 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:44.749422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:44.749471 (+ 49us) server_negotiation.cc:407] Connection header received
0928 06:33:44.750044 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:44.750046 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:44.750058 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:44.750123 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:44.751051 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:44.751557 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:44.752539 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:44.752766 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:44.752854 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:44.754995 (+ 2141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:44.754996 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:44.754997 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:44.755004 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:44.756527 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:44.759092 (+ 2565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:44.759094 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:44.759094 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:44.759157 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:44.759441 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:44.759442 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:44.759442 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:44.759576 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:44.759627 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:44.759992 (+ 365us) server_negotiation.cc:299] Negotiation successful
0928 06:33:44.760071 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210}
I20260928 06:33:44.760335 23314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:44.749171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40365 (local address 127.16.157.65:37379)
0928 06:33:44.749356 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:44.749397 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:44.749940 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:44.750191 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:44.750195 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:44.750759 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:44.750950 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:44.750955 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:44.751658 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:44.751659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:44.752450 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:44.752452 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:44.752866 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:44.752867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:44.752993 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:44.753507 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:44.753516 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:44.754896 (+ 1380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:44.756999 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:44.757007 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:44.757009 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:44.758994 (+ 1985us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:44.759231 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:44.759231 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:44.759232 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:44.759355 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:44.759663 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:44.759670 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:44.759829 (+ 159us) client_negotiation.cc:769] Sending connection context
0928 06:33:44.759956 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 06:33:44.760110 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":101,"threads_started":1}
I20260928 06:33:44.760761 23312 heartbeater.cc:347] Connected to a master server at 127.16.157.126:40365
I20260928 06:33:44.760859 23312 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:44.761035 23312 heartbeater.cc:511] Master 127.16.157.126:40365 requested a full tablet report, sending...
I20260928 06:33:44.761500 23118 ts_manager.cc:194] Registered new tserver with Master: d8d60b2a738a49288570af1432d7c119 (127.16.157.65:36871)
I20260928 06:33:44.762130 23118 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.157.65@KRBTEST.COM'} at 127.16.157.65:37379
WARNING: no policy specified for oryx/127.16.157.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.157.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.157.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/krb5kdc/oryx_127.16.157.66.keytab.
Entry for principal oryx/127.16.157.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/krb5kdc/oryx_127.16.157.66.keytab.
I20260928 06:33:44.784504 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.66:0
--local_ip_for_outbound_sockets=127.16.157.66
--webserver_interface=127.16.157.66
--webserver_port=0
--tserver_master_addrs=127.16.157.126:40365
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/krb5kdc/oryx_127.16.157.66.keytab
--principal=oryx/127.16.157.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:39253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:44.873493 23317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:44.873799 23317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:44.873880 23317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:44.875447 23317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:44.875517 23317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:44.875563 23317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.66
I20260928 06:33:44.877048 23317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:39253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/krb5kdc/oryx_127.16.157.66.keytab
--principal=oryx/127.16.157.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.157.66
--webserver_port=0
--tserver_master_addrs=127.16.157.126:40365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23317
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.66
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:44.877287 23317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:44.877534 23317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:44.882524 23322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:44.882520 23323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:44.882520 23326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:44.882757 23317 server_base.cc:1034] running on GCE node
I20260928 06:33:44.883064 23317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:44.883347 23317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:44.884514 23317 hybrid_clock.cc:648] HybridClock initialized: now 1790577224884497 us; error 34 us; skew 500 ppm
Sep 28 06:33:44 dist-test-slave-rrj5 krb5kdc[23073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577224, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.157.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:44.889554 23317 init.cc:377] Logged in from keytab as oryx/127.16.157.66@KRBTEST.COM (short username oryx)
I20260928 06:33:44.890153 23317 webserver.cc:460] Webserver started at http://127.16.157.66:36103/ using document root <none> and password file <none>
I20260928 06:33:44.890393 23317 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:44.890467 23317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:44.890627 23317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:44.891664 23317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-1/data/instance:
uuid: "ae0e85a2c6794239a27fd60294c0a03f"
format_stamp: "Formatted at 2026-09-28 06:33:44 on dist-test-slave-rrj5"
server_key: "93b8c96a036421cbc2418082ce5c31be"
server_key_iv: "039579f7e28b97bf97275b8946e85a08"
server_key_version: "encryptionkey@0"
I20260928 06:33:44.892023 23317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-1/wal/instance:
uuid: "ae0e85a2c6794239a27fd60294c0a03f"
format_stamp: "Formatted at 2026-09-28 06:33:44 on dist-test-slave-rrj5"
server_key: "93b8c96a036421cbc2418082ce5c31be"
server_key_iv: "039579f7e28b97bf97275b8946e85a08"
server_key_version: "encryptionkey@0"
I20260928 06:33:44.893324 23317 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:33:44.894151 23333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:44.894376 23317 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:33:44.894469 23317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-1/wal
uuid: "ae0e85a2c6794239a27fd60294c0a03f"
format_stamp: "Formatted at 2026-09-28 06:33:44 on dist-test-slave-rrj5"
server_key: "93b8c96a036421cbc2418082ce5c31be"
server_key_iv: "039579f7e28b97bf97275b8946e85a08"
server_key_version: "encryptionkey@0"
I20260928 06:33:44.894532 23317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:44.903941 23317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:44.906191 23317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:44.906333 23317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:44.906556 23317 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:33:44.906899 23317 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:33:44.906948 23317 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:44.906975 23317 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:33:44.906991 23317 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:44.914033 23317 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.66:39405
I20260928 06:33:44.914067 23446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.66:39405 every 8 connection(s)
I20260928 06:33:44.914414 23317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
I20260928 06:33:44.920352 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 23317
I20260928 06:33:44.920490 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-1/wal/instance
I20260928 06:33:44.920679 17013 external_mini_cluster.cc:1428] Setting key b992e340294e0be1e86baaa8e4761b94
Sep 28 06:33:44 dist-test-slave-rrj5 krb5kdc[23073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577224, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.157.66@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM
I20260928 06:33:44.924496 23159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:44.915131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.66:57761 (local address 127.16.157.126:40365)
0928 06:33:44.915196 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:44.915198 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:44.915399 (+ 201us) server_negotiation.cc:407] Connection header received
0928 06:33:44.915877 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:44.915878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:44.915890 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:44.915963 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:44.916885 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:44.917627 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:44.918603 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:44.918891 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:44.918966 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:44.921272 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:44.921274 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:44.921274 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:44.921282 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:44.922761 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:44.923368 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:44.923370 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:44.923370 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:44.923452 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:44.923745 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:44.923746 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:44.923746 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:44.923885 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:44.923952 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:44.924315 (+ 363us) server_negotiation.cc:299] Negotiation successful
0928 06:33:44.924395 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 06:33:44.925006 23449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:44.915063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40365 (local address 127.16.157.66:57761)
0928 06:33:44.915267 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:44.915308 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:44.915785 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:44.916034 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:44.916038 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:44.916605 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:44.916787 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:44.916792 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:44.917710 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:44.917711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:44.918518 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:44.918520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:44.918975 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:44.918976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:44.919072 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:44.919602 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:44.919609 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:44.921161 (+ 1552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:44.922860 (+ 1699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:44.922867 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:44.922869 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:44.923282 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:44.923513 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:44.923514 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:44.923515 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:44.923662 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:44.923961 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:44.923966 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:44.924142 (+ 176us) client_negotiation.cc:769] Sending connection context
0928 06:33:44.924268 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 06:33:44.924453 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":113,"threads_started":1}
I20260928 06:33:44.925511 23447 heartbeater.cc:347] Connected to a master server at 127.16.157.126:40365
I20260928 06:33:44.925632 23447 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:44.925837 23447 heartbeater.cc:511] Master 127.16.157.126:40365 requested a full tablet report, sending...
I20260928 06:33:44.926285 23118 ts_manager.cc:194] Registered new tserver with Master: ae0e85a2c6794239a27fd60294c0a03f (127.16.157.66:39405)
I20260928 06:33:44.926851 23118 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.157.66@KRBTEST.COM'} at 127.16.157.66:57761
WARNING: no policy specified for oryx/127.16.157.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.157.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.157.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/krb5kdc/oryx_127.16.157.67.keytab.
Entry for principal oryx/127.16.157.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/krb5kdc/oryx_127.16.157.67.keytab.
I20260928 06:33:44.948652 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.67:0
--local_ip_for_outbound_sockets=127.16.157.67
--webserver_interface=127.16.157.67
--webserver_port=0
--tserver_master_addrs=127.16.157.126:40365
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/krb5kdc/oryx_127.16.157.67.keytab
--principal=oryx/127.16.157.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:39253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:45.032234 23452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:45.032480 23452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:45.032521 23452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:45.033803 23452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:45.033864 23452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:45.033912 23452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.67
I20260928 06:33:45.035410 23452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:39253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/krb5kdc/oryx_127.16.157.67.keytab
--principal=oryx/127.16.157.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.157.67
--webserver_port=0
--tserver_master_addrs=127.16.157.126:40365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23452
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.67
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:45.035671 23452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:45.035912 23452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:45.041275 23458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:45.041352 23457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:45.041442 23452 server_base.cc:1034] running on GCE node
W20260928 06:33:45.041688 23460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:45.041903 23452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:45.042167 23452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:45.043371 23452 hybrid_clock.cc:648] HybridClock initialized: now 1790577225043347 us; error 39 us; skew 500 ppm
Sep 28 06:33:45 dist-test-slave-rrj5 krb5kdc[23073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577225, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.157.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:45.046034 23452 init.cc:377] Logged in from keytab as oryx/127.16.157.67@KRBTEST.COM (short username oryx)
I20260928 06:33:45.046489 23452 webserver.cc:460] Webserver started at http://127.16.157.67:46135/ using document root <none> and password file <none>
I20260928 06:33:45.046723 23452 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:45.046797 23452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:45.046941 23452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:45.048027 23452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-2/data/instance:
uuid: "6cec1f1c72de4f28a819e930d1f014b5"
format_stamp: "Formatted at 2026-09-28 06:33:45 on dist-test-slave-rrj5"
server_key: "5a170e70375f49610fc6f368d8496ef1"
server_key_iv: "93c71114e136000ff0853b81f32e9e21"
server_key_version: "encryptionkey@0"
I20260928 06:33:45.048413 23452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-2/wal/instance:
uuid: "6cec1f1c72de4f28a819e930d1f014b5"
format_stamp: "Formatted at 2026-09-28 06:33:45 on dist-test-slave-rrj5"
server_key: "5a170e70375f49610fc6f368d8496ef1"
server_key_iv: "93c71114e136000ff0853b81f32e9e21"
server_key_version: "encryptionkey@0"
I20260928 06:33:45.049813 23452 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:33:45.050631 23468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:45.050830 23452 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:45.050931 23452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-2/wal
uuid: "6cec1f1c72de4f28a819e930d1f014b5"
format_stamp: "Formatted at 2026-09-28 06:33:45 on dist-test-slave-rrj5"
server_key: "5a170e70375f49610fc6f368d8496ef1"
server_key_iv: "93c71114e136000ff0853b81f32e9e21"
server_key_version: "encryptionkey@0"
I20260928 06:33:45.051011 23452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:45.059252 23452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:45.061491 23452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:45.061676 23452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:45.061898 23452 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:33:45.062222 23452 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:33:45.062273 23452 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:45.062304 23452 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:33:45.062331 23452 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:45.070348 23452 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.67:33283
I20260928 06:33:45.070432 23581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.67:33283 every 8 connection(s)
I20260928 06:33:45.070747 23452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
I20260928 06:33:45.073827 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 23452
I20260928 06:33:45.073931 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-2/wal/instance
I20260928 06:33:45.074123 17013 external_mini_cluster.cc:1428] Setting key 703d245a1d75634b25ecd942f26344db
Sep 28 06:33:45 dist-test-slave-rrj5 krb5kdc[23073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577225, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.157.67@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM
I20260928 06:33:45.082319 23159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:45.071564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:40483 (local address 127.16.157.126:40365)
0928 06:33:45.071637 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:45.071640 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:45.071856 (+ 216us) server_negotiation.cc:407] Connection header received
0928 06:33:45.072659 (+ 803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:45.072660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:45.072673 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:45.072775 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:45.074117 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.074884 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.076423 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.076617 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.076683 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:45.079348 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:45.079350 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:45.079350 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:45.079358 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:45.080821 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:45.081257 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:45.081258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:45.081259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:45.081327 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:45.081642 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:45.081643 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:45.081643 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:45.081762 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:45.081835 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:45.082132 (+ 297us) server_negotiation.cc:299] Negotiation successful
0928 06:33:45.082227 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 06:33:45.082656 23584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:45.071486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40365 (local address 127.16.157.67:40483)
0928 06:33:45.071717 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:45.071761 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:45.072535 (+ 774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:45.072835 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:45.072839 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:45.073612 (+ 773us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:45.074003 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:45.074010 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.074980 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.074981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:45.076306 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:45.076309 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.076719 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.076720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:45.076886 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:45.077469 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:45.077480 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:45.079227 (+ 1747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:45.080925 (+ 1698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:45.080933 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:45.080935 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:45.081174 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:45.081386 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:45.081387 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:45.081387 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:45.081516 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:45.081839 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:45.081846 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:45.081992 (+ 146us) client_negotiation.cc:769] Sending connection context
0928 06:33:45.082095 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 06:33:45.082262 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":128,"threads_started":1}
I20260928 06:33:45.083174 23582 heartbeater.cc:347] Connected to a master server at 127.16.157.126:40365
I20260928 06:33:45.083305 23582 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:45.083530 23582 heartbeater.cc:511] Master 127.16.157.126:40365 requested a full tablet report, sending...
I20260928 06:33:45.083930 23118 ts_manager.cc:194] Registered new tserver with Master: 6cec1f1c72de4f28a819e930d1f014b5 (127.16.157.67:33283)
I20260928 06:33:45.084386 23118 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.157.67@KRBTEST.COM'} at 127.16.157.67:40483
I20260928 06:33:45.086124 17013 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.16.157.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.157.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.157.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/krb5kdc/oryx_127.16.157.125.keytab.
Entry for principal oryx/127.16.157.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/krb5kdc/oryx_127.16.157.125.keytab.
I20260928 06:33:45.112536 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.125:40377
--webserver_interface=127.16.157.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/krb5kdc/oryx_127.16.157.125.keytab
--principal=oryx/127.16.157.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:39253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.157.126:40365,127.16.157.125:40377
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:45.198124 23587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:45.198352 23587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:45.198390 23587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:45.199573 23587 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:33:45.199635 23587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:45.199658 23587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:45.199680 23587 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:33:45.199709 23587 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:33:45.201108 23587 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:39253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.157.126:40365,127.16.157.125:40377
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/krb5kdc/oryx_127.16.157.125.keytab
--principal=oryx/127.16.157.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.125:40377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.157.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
W20260928 06:33:45.201236 23587 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.16.157.126:40365,127.16.157.125:40377'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 06:33:45.201391 23587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:45.201619 23587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:45.207166 23593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:45.207135 23592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:45.207407 23595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:45.208113 23587 server_base.cc:1034] running on GCE node
I20260928 06:33:45.208351 23587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:45.208667 23587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:45.209824 23587 hybrid_clock.cc:648] HybridClock initialized: now 1790577225209812 us; error 31 us; skew 500 ppm
Sep 28 06:33:45 dist-test-slave-rrj5 krb5kdc[23073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577225, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.157.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:45.212262 23587 init.cc:377] Logged in from keytab as oryx/127.16.157.125@KRBTEST.COM (short username oryx)
I20260928 06:33:45.212687 23587 webserver.cc:460] Webserver started at http://127.16.157.125:35589/ using document root <none> and password file <none>
I20260928 06:33:45.212908 23587 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:45.212972 23587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:45.213102 23587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:45.214452 23587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/master-1/data/instance:
uuid: "9728b0a476874f8d84efd61d0b15a7eb"
format_stamp: "Formatted at 2026-09-28 06:33:45 on dist-test-slave-rrj5"
server_key: "967e888a2e9d71c277fec64a589ce7ff"
server_key_iv: "0a994b5d14fab1c14019eb37c16d412c"
server_key_version: "encryptionkey@0"
I20260928 06:33:45.214906 23587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/master-1/wal/instance:
uuid: "9728b0a476874f8d84efd61d0b15a7eb"
format_stamp: "Formatted at 2026-09-28 06:33:45 on dist-test-slave-rrj5"
server_key: "967e888a2e9d71c277fec64a589ce7ff"
server_key_iv: "0a994b5d14fab1c14019eb37c16d412c"
server_key_version: "encryptionkey@0"
I20260928 06:33:45.216492 23587 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:33:45.217648 23602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:45.217829 23587 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:45.217926 23587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/master-1/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/master-1/wal
uuid: "9728b0a476874f8d84efd61d0b15a7eb"
format_stamp: "Formatted at 2026-09-28 06:33:45 on dist-test-slave-rrj5"
server_key: "967e888a2e9d71c277fec64a589ce7ff"
server_key_iv: "0a994b5d14fab1c14019eb37c16d412c"
server_key_version: "encryptionkey@0"
I20260928 06:33:45.218011 23587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:45.228370 23587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:45.230997 23587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:45.231153 23587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:45.235666 23587 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.125:40377
I20260928 06:33:45.235760 23655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.125:40377 every 8 connection(s)
I20260928 06:33:45.236081 23587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/master-1/data/info.pb
I20260928 06:33:45.237299 23656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:45.237883 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 23587
I20260928 06:33:45.239944 23656 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40365 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:33:45 dist-test-slave-rrj5 krb5kdc[23073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.157.125@KRBTEST.COM
Sep 28 06:33:45 dist-test-slave-rrj5 krb5kdc[23073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577225, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.157.125@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM
I20260928 06:33:45.249333 23659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:45.238479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33826 (local address 127.16.157.125:40377)
0928 06:33:45.238649 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:45.238658 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:45.238712 (+ 54us) server_negotiation.cc:407] Connection header received
0928 06:33:45.239150 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:45.239157 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:45.239203 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:45.239357 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:45.239974 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.241263 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.242320 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.242824 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.242946 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:45.245758 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:45.245766 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:45.245767 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:45.245773 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:45.247621 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:45.248073 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:45.248076 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:45.248077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:45.248177 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:45.248400 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:45.248402 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:45.248402 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:45.248717 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:45.248878 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:45.248977 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 06:33:45.249114 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":74,"threads_started":1}
I20260928 06:33:45.249778 23159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:45.240640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48770 (local address 127.16.157.126:40365)
0928 06:33:45.240701 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:45.240703 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:45.240898 (+ 195us) server_negotiation.cc:407] Connection header received
0928 06:33:45.240967 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:45.240968 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:45.240979 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:45.241029 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:45.241615 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.242351 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.243557 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.243887 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.243970 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:45.246663 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:45.246664 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:45.246665 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:45.246672 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:45.248316 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:45.248712 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:45.248713 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:45.248714 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:45.248772 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:45.248985 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:45.248986 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:45.248986 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:45.249185 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:45.249234 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:45.249565 (+ 331us) server_negotiation.cc:299] Negotiation successful
0928 06:33:45.249681 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 06:33:45.250128 23660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:45.240589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40365 (local address 127.0.0.1:48770)
0928 06:33:45.240778 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:45.240817 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:45.240891 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:45.241070 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:45.241073 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:45.241401 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:45.241487 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:45.241489 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.242469 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.242470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:45.243459 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:45.243462 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.243992 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.243993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:45.244078 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:45.244623 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:45.244634 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:45.246528 (+ 1894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:45.248407 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:45.248413 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:45.248416 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:45.248629 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:45.248846 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:45.248847 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:45.248847 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:45.248911 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:45.249343 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:45.249348 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:45.249422 (+ 74us) client_negotiation.cc:769] Sending connection context
0928 06:33:45.249880 (+ 458us) client_negotiation.cc:240] Negotiation successful
0928 06:33:45.250012 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":97,"threads_started":1}
I20260928 06:33:45.250638 23656 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.157.125" port: 40377 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:33:45 dist-test-slave-rrj5 krb5kdc[23073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577225, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.157.125@KRBTEST.COM for oryx/127.16.157.125@KRBTEST.COM
I20260928 06:33:45.256822 23660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:45.250921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.125:40377 (local address 127.0.0.1:33840)
0928 06:33:45.250974 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:45.250987 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:45.251034 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:45.251155 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:45.251155 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:45.251290 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:45.251317 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:45.251318 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.251957 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.251957 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:45.252438 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:45.252439 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.252797 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.252797 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:45.252856 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:45.253298 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:45.253302 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:45.254408 (+ 1106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:45.255847 (+ 1439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:45.255848 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:45.255848 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:45.256002 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:45.256198 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:45.256199 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:45.256199 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:45.256249 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:45.256489 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:45.256490 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:45.256525 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 06:33:45.256628 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 06:33:45.256719 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 06:33:45.256886 23659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:45.250967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33840 (local address 127.16.157.125:40377)
0928 06:33:45.251018 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:45.251020 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:45.251028 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:33:45.251089 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:45.251090 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:45.251098 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:45.251151 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:45.251396 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.251870 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.252524 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.252713 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.252799 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:45.254502 (+ 1703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:45.254504 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:45.254504 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:45.254510 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:45.255737 (+ 1227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:45.256081 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:45.256082 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:45.256083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:45.256129 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:45.256324 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:45.256325 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:45.256325 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:45.256427 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:45.256496 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:45.256650 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 06:33:45.256767 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":37}
I20260928 06:33:45.258481 23656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb: Bootstrap starting.
I20260928 06:33:45.259305 23656 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:45.259594 23656 log.cc:824] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:45.260344 23656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb: No bootstrap required, opened a new log
I20260928 06:33:45.262041 23656 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a9edcc4ed794902a9d229837466efd2" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40365 } } peers { permanent_uuid: "9728b0a476874f8d84efd61d0b15a7eb" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 40377 } }
I20260928 06:33:45.262153 23656 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:45.262174 23656 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9728b0a476874f8d84efd61d0b15a7eb, State: Initialized, Role: FOLLOWER
I20260928 06:33:45.262295 23656 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a9edcc4ed794902a9d229837466efd2" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40365 } } peers { permanent_uuid: "9728b0a476874f8d84efd61d0b15a7eb" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 40377 } }
I20260928 06:33:45.262621 23656 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:33:45.262593 23662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a9edcc4ed794902a9d229837466efd2" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40365 } } peers { permanent_uuid: "9728b0a476874f8d84efd61d0b15a7eb" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 40377 } } }
I20260928 06:33:45.262741 23662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb [sys.catalog]: This master's current role is: FOLLOWER
W20260928 06:33:45.264261 23673 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:33:45.264314 23673 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:33:45.265897 23587 master_runner.cc:428] Detected that this master 9728b0a476874f8d84efd61d0b15a7eb is joining an existing cluster
I20260928 06:33:45.265957 23587 master_runner.cc:432] Initiating AddMaster RPC to add 127.16.157.125:40377
I20260928 06:33:45.273435 23659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:45.267146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33856 (local address 127.16.157.125:40377)
0928 06:33:45.267184 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:45.267186 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:45.267398 (+ 212us) server_negotiation.cc:407] Connection header received
0928 06:33:45.267490 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:45.267491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:45.267507 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:45.267591 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:45.267969 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.268457 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.269198 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.269389 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.269467 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:45.270585 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:45.270585 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:45.270586 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:45.270591 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:45.272164 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:45.272536 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:45.272537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:45.272537 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:45.272582 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:45.272761 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:45.272762 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:45.272762 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:45.272938 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:45.272984 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:45.273184 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 06:33:45.273321 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":76,"server-negotiator.queue_time_us":20}
I20260928 06:33:45.274004 23680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:45.267228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.125:40377 (local address 127.0.0.1:33856)
0928 06:33:45.267352 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:45.267360 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:45.267406 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:45.267594 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:45.267595 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:45.267793 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:45.267891 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:45.267893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.268545 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.268547 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:45.269117 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:45.269119 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.269472 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.269473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:45.269529 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:45.270076 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:45.270082 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:45.270505 (+ 423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:45.272248 (+ 1743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:45.272249 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:45.272249 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:45.272468 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:45.272642 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:45.272643 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:45.272643 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:45.272693 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:45.273024 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:45.273024 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:45.273063 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 06:33:45.273159 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 06:33:45.273865 (+ 706us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":67,"threads_started":1}
I20260928 06:33:45.276487 23159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:45.269746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48784 (local address 127.16.157.126:40365)
0928 06:33:45.269814 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:45.269816 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:45.269824 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:33:45.269845 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:45.269846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:45.269856 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:45.269925 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:45.270377 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.271138 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.272322 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.272649 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.272690 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:45.273759 (+ 1069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:45.273760 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:45.273761 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:45.273767 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:45.275356 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:45.275720 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:45.275721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:45.275722 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:45.275776 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:45.275987 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:45.275988 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:45.275988 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:45.276101 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:45.276176 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:45.276324 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 06:33:45.276398 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 06:33:45.276518 23679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:45.266890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40365 (local address 127.0.0.1:48784)
0928 06:33:45.267228 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:45.267239 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:45.267301 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:45.269943 (+ 2642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:45.269944 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:45.270221 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:45.270264 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:45.270266 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.271237 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.271237 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:45.272215 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:45.272216 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.272767 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.272768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:45.272838 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:45.273246 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:45.273250 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:45.273667 (+ 417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:45.275461 (+ 1794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:45.275462 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:45.275462 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:45.275638 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:45.275853 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:45.275854 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:45.275854 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:45.275917 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:45.276167 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:45.276168 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:45.276210 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 06:33:45.276298 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 06:33:45.276399 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":85,"threads_started":1}
Sep 28 06:33:45 dist-test-slave-rrj5 krb5kdc[23073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577224, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.157.126@KRBTEST.COM for oryx/127.16.157.125@KRBTEST.COM
I20260928 06:33:45.286000 23659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:45.278747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33858 (local address 127.16.157.125:40377)
0928 06:33:45.278785 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:45.278787 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:45.278976 (+ 189us) server_negotiation.cc:407] Connection header received
0928 06:33:45.279060 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:45.279061 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:45.279071 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:45.279119 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:45.279640 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.280086 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.280866 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.281065 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.281139 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:45.283213 (+ 2074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:45.283214 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:45.283214 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:45.283221 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:45.284656 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:45.285072 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:45.285073 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:45.285073 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:45.285127 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:45.285379 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:45.285380 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:45.285380 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:45.285516 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:45.285606 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:45.285828 (+ 222us) server_negotiation.cc:299] Negotiation successful
0928 06:33:45.285903 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 06:33:45.286028 23681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:45.278722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.125:40377 (local address 127.0.0.1:33858)
0928 06:33:45.278859 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:45.278906 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:45.278982 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:45.279254 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:45.279259 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:45.279513 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:45.279558 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:45.279559 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.280169 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.280170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:45.280769 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:45.280770 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.281140 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.281141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:45.281226 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:45.281819 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:45.281824 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:45.283123 (+ 1299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:45.284738 (+ 1615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:45.284786 (+ 48us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:45.284788 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:45.284994 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:45.285201 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:45.285202 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:45.285203 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:45.285262 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:45.285623 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:45.285630 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:45.285689 (+ 59us) client_negotiation.cc:769] Sending connection context
0928 06:33:45.285797 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 06:33:45.285916 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":88,"threads_started":1}
I20260928 06:33:45.286899 23118 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.16.157.125:40377: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "9728b0a476874f8d84efd61d0b15a7eb"
member_type: NON_VOTER
last_known_addr {
host: "127.16.157.125"
port: 40377
}
attrs {
promote: true
}
}
dest_uuid: "3a9edcc4ed794902a9d229837466efd2"
cas_config_opid_index: -1
I20260928 06:33:45.287227 23118 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a9edcc4ed794902a9d229837466efd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3a9edcc4ed794902a9d229837466efd2" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40365 } } peers { permanent_uuid: "9728b0a476874f8d84efd61d0b15a7eb" member_type: NON_VOTER last_known_addr { host: "127.16.157.125" port: 40377 } attrs { promote: true } }
I20260928 06:33:45.287719 23683 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3a9edcc4ed794902a9d229837466efd2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 9728b0a476874f8d84efd61d0b15a7eb (127.16.157.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3a9edcc4ed794902a9d229837466efd2" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40365 } } peers { permanent_uuid: "9728b0a476874f8d84efd61d0b15a7eb" member_type: NON_VOTER last_known_addr { host: "127.16.157.125" port: 40377 } attrs { promote: true } } }
I20260928 06:33:45.288081 23630 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:45.288844 23630 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb [term 1 FOLLOWER]: Refusing update from remote peer 3a9edcc4ed794902a9d229837466efd2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 06:33:45.289183 23682 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3a9edcc4ed794902a9d229837466efd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9728b0a476874f8d84efd61d0b15a7eb" member_type: NON_VOTER last_known_addr { host: "127.16.157.125" port: 40377 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:33:45.291198 23630 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 9728b0a476874f8d84efd61d0b15a7eb (127.16.157.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3a9edcc4ed794902a9d229837466efd2" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40365 } } peers { permanent_uuid: "9728b0a476874f8d84efd61d0b15a7eb" member_type: NON_VOTER last_known_addr { host: "127.16.157.125" port: 40377 } attrs { promote: true } } }
I20260928 06:33:45.292131 23690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3a9edcc4ed794902a9d229837466efd2" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3a9edcc4ed794902a9d229837466efd2" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40365 } } peers { permanent_uuid: "9728b0a476874f8d84efd61d0b15a7eb" member_type: NON_VOTER last_known_addr { host: "127.16.157.125" port: 40377 } attrs { promote: true } } }
I20260928 06:33:45.292296 23690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb [sys.catalog]: This master's current role is: LEARNER
I20260928 06:33:45.292553 23686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3a9edcc4ed794902a9d229837466efd2" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3a9edcc4ed794902a9d229837466efd2" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40365 } } peers { permanent_uuid: "9728b0a476874f8d84efd61d0b15a7eb" member_type: NON_VOTER last_known_addr { host: "127.16.157.125" port: 40377 } attrs { promote: true } } }
I20260928 06:33:45.292632 23686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb [sys.catalog]: This master's current role is: LEARNER
I20260928 06:33:45.292737 23662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a9edcc4ed794902a9d229837466efd2. Latest consensus state: current_term: 1 leader_uuid: "3a9edcc4ed794902a9d229837466efd2" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3a9edcc4ed794902a9d229837466efd2" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40365 } } peers { permanent_uuid: "9728b0a476874f8d84efd61d0b15a7eb" member_type: NON_VOTER last_known_addr { host: "127.16.157.125" port: 40377 } attrs { promote: true } } }
I20260928 06:33:45.292809 23662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb [sys.catalog]: This master's current role is: LEARNER
I20260928 06:33:45.293020 23683 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.16.157.125:40377
I20260928 06:33:45.293231 23587 master.cc:537] Master@127.16.157.125:40377 shutting down...
I20260928 06:33:45.293244 23682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a9edcc4ed794902a9d229837466efd2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "3a9edcc4ed794902a9d229837466efd2" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3a9edcc4ed794902a9d229837466efd2" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40365 } } peers { permanent_uuid: "9728b0a476874f8d84efd61d0b15a7eb" member_type: NON_VOTER last_known_addr { host: "127.16.157.125" port: 40377 } attrs { promote: true } } }
I20260928 06:33:45.293334 23682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a9edcc4ed794902a9d229837466efd2 [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:45.293540 23682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a9edcc4ed794902a9d229837466efd2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3a9edcc4ed794902a9d229837466efd2" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3a9edcc4ed794902a9d229837466efd2" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40365 } } peers { permanent_uuid: "9728b0a476874f8d84efd61d0b15a7eb" member_type: NON_VOTER last_known_addr { host: "127.16.157.125" port: 40377 } attrs { promote: true } } }
I20260928 06:33:45.293643 23682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a9edcc4ed794902a9d229837466efd2 [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:45.295962 23587 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb [term 1 LEARNER]: Raft consensus shutting down.
I20260928 06:33:45.296090 23587 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb [term 1 LEARNER]: Raft consensus is shut down!
I20260928 06:33:45.296136 23587 tablet_replica.cc:330] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb: stopping tablet replica
I20260928 06:33:45.309242 23587 master.cc:552] Master@127.16.157.125:40377 shutdown complete.
I20260928 06:33:45.309312 23587 master_runner.cc:305] Clearing existing system tablet
I20260928 06:33:45.309459 23587 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:45.309520 23587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:45.310181 23587 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:45.310664 23694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:45.310788 23587 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:45.310839 23587 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:45.310868 23587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/master-1/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/master-1/wal
uuid: "9728b0a476874f8d84efd61d0b15a7eb"
format_stamp: "Formatted at 2026-09-28 06:33:45 on dist-test-slave-rrj5"
server_key: "967e888a2e9d71c277fec64a589ce7ff"
server_key_iv: "0a994b5d14fab1c14019eb37c16d412c"
server_key_version: "encryptionkey@0"
I20260928 06:33:45.311493 23587 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:33:45.312930 23587 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 06:33:45.313000 23587 log.cc:1196] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb: Deleting WAL directory at /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 06:33:45.313231 23587 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb: Deleting consensus metadata
I20260928 06:33:45.313370 23587 master_runner.cc:315] Copying system tablet from 127.16.157.126:40365
I20260928 06:33:45.314092 23587 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb: tablet copy: Beginning tablet copy session from remote peer at address 127.16.157.126:40365
I20260928 06:33:45.322101 23159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:45.314709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48788 (local address 127.16.157.126:40365)
0928 06:33:45.314761 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:45.314764 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:45.314773 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:33:45.314799 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:45.314799 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:45.314810 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:45.314862 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:45.315575 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.316427 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.317521 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.317966 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.318057 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:45.319124 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:45.319125 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:45.319125 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:45.319133 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:45.320794 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:45.321207 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:45.321209 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:45.321209 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:45.321276 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:45.321499 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:45.321500 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:45.321501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:45.321676 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:45.321747 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:45.321931 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 06:33:45.322012 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 06:33:45.322113 23700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:45.314425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40365 (local address 127.0.0.1:48788)
0928 06:33:45.314548 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:45.314561 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:45.314634 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:45.314902 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:45.314903 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:45.315110 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:45.315468 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:45.315470 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.316528 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.316529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:45.317420 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:45.317423 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.318069 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.318070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:45.318146 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:45.318613 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:45.318618 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:45.319024 (+ 406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:45.320888 (+ 1864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:45.320889 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:45.320889 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:45.321122 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:45.321352 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:45.321353 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:45.321353 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:45.321441 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:45.321746 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:45.321747 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:45.321807 (+ 60us) client_negotiation.cc:769] Sending connection context
0928 06:33:45.321885 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 06:33:45.321992 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":67,"threads_started":1}
I20260928 06:33:45.322444 23138 tablet_copy_service.cc:140] P 3a9edcc4ed794902a9d229837466efd2: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 9728b0a476874f8d84efd61d0b15a7eb ({username='oryx', principal='oryx/127.16.157.125@KRBTEST.COM'} at 127.0.0.1:48788)
I20260928 06:33:45.322556 23138 tablet_copy_service.cc:161] P 3a9edcc4ed794902a9d229837466efd2: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 9728b0a476874f8d84efd61d0b15a7eb at {username='oryx', principal='oryx/127.16.157.125@KRBTEST.COM'} at 127.0.0.1:48788: session id = 9728b0a476874f8d84efd61d0b15a7eb-00000000000000000000000000000000
I20260928 06:33:45.323406 23138 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 3a9edcc4ed794902a9d229837466efd2: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 06:33:45.324052 23587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 06:33:45.325369 23587 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb: tablet copy: Starting download of 0 data blocks...
I20260928 06:33:45.325529 23587 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb: tablet copy: Starting download of 1 WAL segments...
I20260928 06:33:45.326496 23587 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 06:33:45.327406 23138 tablet_copy_service.cc:342] P 3a9edcc4ed794902a9d229837466efd2: Request end of tablet copy session 9728b0a476874f8d84efd61d0b15a7eb-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.16.157.125@KRBTEST.COM'} at 127.0.0.1:48788
I20260928 06:33:45.327462 23138 tablet_copy_service.cc:433] P 3a9edcc4ed794902a9d229837466efd2: ending tablet copy session 9728b0a476874f8d84efd61d0b15a7eb-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 9728b0a476874f8d84efd61d0b15a7eb
I20260928 06:33:45.330544 23587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:45.331658 23701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:45.331894 23587 server_base.cc:1034] running on GCE node
W20260928 06:33:45.332047 23704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:45.332193 23702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:45.332423 23587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:45.332589 23587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:45.333698 23587 hybrid_clock.cc:648] HybridClock initialized: now 1790577225333690 us; error 23 us; skew 500 ppm
Sep 28 06:33:45 dist-test-slave-rrj5 krb5kdc[23073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577225, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.157.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:45.335234 23587 init.cc:377] Logged in from keytab as oryx/127.16.157.125@KRBTEST.COM (short username oryx)
I20260928 06:33:45.335502 23587 webserver.cc:460] Webserver started at http://127.16.157.125:35773/ using document root <none> and password file <none>
I20260928 06:33:45.335615 23587 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:45.335649 23587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:45.336347 23587 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:33:45.336949 23711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:45.337116 23587 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:33:45.337178 23587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/master-1/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/master-1/wal
uuid: "9728b0a476874f8d84efd61d0b15a7eb"
format_stamp: "Formatted at 2026-09-28 06:33:45 on dist-test-slave-rrj5"
server_key: "967e888a2e9d71c277fec64a589ce7ff"
server_key_iv: "0a994b5d14fab1c14019eb37c16d412c"
server_key_version: "encryptionkey@0"
I20260928 06:33:45.337230 23587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:45.370697 23587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:45.372900 23587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:45.376457 23587 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.125:40377
I20260928 06:33:45.376507 23764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.125:40377 every 8 connection(s)
I20260928 06:33:45.376806 23587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/master-1/data/info.pb
I20260928 06:33:45.377532 23765 sys_catalog.cc:263] Verifying existing consensus state
I20260928 06:33:45.378121 23765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb: Bootstrap starting.
I20260928 06:33:45.380381 23765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:33:45.380589 23765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb: Bootstrap complete.
I20260928 06:33:45.380781 23765 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3a9edcc4ed794902a9d229837466efd2" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40365 } } peers { permanent_uuid: "9728b0a476874f8d84efd61d0b15a7eb" member_type: NON_VOTER last_known_addr { host: "127.16.157.125" port: 40377 } attrs { promote: true } }
I20260928 06:33:45.380867 23765 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9728b0a476874f8d84efd61d0b15a7eb, State: Initialized, Role: LEARNER
I20260928 06:33:45.380903 23765 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3a9edcc4ed794902a9d229837466efd2" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40365 } } peers { permanent_uuid: "9728b0a476874f8d84efd61d0b15a7eb" member_type: NON_VOTER last_known_addr { host: "127.16.157.125" port: 40377 } attrs { promote: true } }
I20260928 06:33:45.381080 23765 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:33:45.381074 23768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3a9edcc4ed794902a9d229837466efd2" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40365 } } peers { permanent_uuid: "9728b0a476874f8d84efd61d0b15a7eb" member_type: NON_VOTER last_known_addr { host: "127.16.157.125" port: 40377 } attrs { promote: true } } }
I20260928 06:33:45.381297 23768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb [sys.catalog]: This master's current role is: LEARNER
I20260928 06:33:45.382555 23779 catalog_manager.cc:1259] Loaded cluster ID: fc73c8eec1414312b3fe0164efd101a8
I20260928 06:33:45.382606 23779 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb: loading cluster ID for follower catalog manager: success
I20260928 06:33:45.383736 23779 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb: acquiring CA information for follower catalog manager: success
I20260928 06:33:45.384130 23779 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 06:33:45.410578 23780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:45.402667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33884 (local address 127.16.157.125:40377)
0928 06:33:45.402938 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:45.402941 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:45.402951 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:33:45.402979 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:45.402980 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:45.402992 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:45.403085 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:45.403779 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.404648 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.405836 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.406141 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.406216 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:45.407672 (+ 1456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:45.407674 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:45.407675 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:45.407682 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:45.409291 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:45.409689 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:45.409690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:45.409691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:45.409756 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:45.409955 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:45.409955 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:45.409956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:45.410087 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:45.410181 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:45.410295 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 06:33:45.410374 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":60,"threads_started":1}
WARNING: no policy specified for oryx/127.16.157.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.157.124@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.157.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/krb5kdc/oryx_127.16.157.124.keytab.
Entry for principal oryx/127.16.157.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/krb5kdc/oryx_127.16.157.124.keytab.
I20260928 06:33:45.438545 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.124:44455
--webserver_interface=127.16.157.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/krb5kdc/oryx_127.16.157.124.keytab
--principal=oryx/127.16.157.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:39253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.157.126:40365,127.16.157.125:40377,127.16.157.124:44455
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:45.525341 23783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:45.525560 23783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:45.525619 23783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:45.526779 23783 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:33:45.526846 23783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:45.526870 23783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:45.526889 23783 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:33:45.526921 23783 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:33:45.528414 23783 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:39253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.157.126:40365,127.16.157.125:40377,127.16.157.124:44455
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/krb5kdc/oryx_127.16.157.124.keytab
--principal=oryx/127.16.157.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.124:44455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.157.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:45.528671 23783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:45.528903 23783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:45.533816 23791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:45.533839 23788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:45.533828 23789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:45.534092 23783 server_base.cc:1034] running on GCE node
I20260928 06:33:45.534297 23783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:45.534653 23783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:45.535805 23783 hybrid_clock.cc:648] HybridClock initialized: now 1790577225535795 us; error 27 us; skew 500 ppm
Sep 28 06:33:45 dist-test-slave-rrj5 krb5kdc[23073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577225, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.157.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:45.538584 23783 init.cc:377] Logged in from keytab as oryx/127.16.157.124@KRBTEST.COM (short username oryx)
I20260928 06:33:45.539085 23783 webserver.cc:460] Webserver started at http://127.16.157.124:42799/ using document root <none> and password file <none>
I20260928 06:33:45.539340 23783 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:45.539397 23783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:45.539523 23783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:45.540828 23783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/master-2/data/instance:
uuid: "61915ec9e9cd43f8b944b77b8ba364f1"
format_stamp: "Formatted at 2026-09-28 06:33:45 on dist-test-slave-rrj5"
server_key: "c6465dca8fb68e2088ec45ced64cab34"
server_key_iv: "2110e4274a754df2344634cca4ac03de"
server_key_version: "encryptionkey@0"
I20260928 06:33:45.541230 23783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/master-2/wal/instance:
uuid: "61915ec9e9cd43f8b944b77b8ba364f1"
format_stamp: "Formatted at 2026-09-28 06:33:45 on dist-test-slave-rrj5"
server_key: "c6465dca8fb68e2088ec45ced64cab34"
server_key_iv: "2110e4274a754df2344634cca4ac03de"
server_key_version: "encryptionkey@0"
I20260928 06:33:45.542815 23783 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260928 06:33:45.543676 23798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:45.543861 23783 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:33:45.543990 23783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/master-2/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/master-2/wal
uuid: "61915ec9e9cd43f8b944b77b8ba364f1"
format_stamp: "Formatted at 2026-09-28 06:33:45 on dist-test-slave-rrj5"
server_key: "c6465dca8fb68e2088ec45ced64cab34"
server_key_iv: "2110e4274a754df2344634cca4ac03de"
server_key_version: "encryptionkey@0"
I20260928 06:33:45.544070 23783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:45.563513 23783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:45.565716 23783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:45.565882 23783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:45.570425 23783 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.124:44455
I20260928 06:33:45.570469 23850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.124:44455 every 8 connection(s)
I20260928 06:33:45.570835 23783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/master-2/data/info.pb
I20260928 06:33:45.571964 23851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:45.573983 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 23783
I20260928 06:33:45.573904 23851 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40365 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:33:45 dist-test-slave-rrj5 krb5kdc[23073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577225, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.157.124@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM
Sep 28 06:33:45 dist-test-slave-rrj5 krb5kdc[23073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.157.124@KRBTEST.COM
I20260928 06:33:45.584450 23159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:45.574730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48792 (local address 127.16.157.126:40365)
0928 06:33:45.574782 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:45.574784 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:45.575032 (+ 248us) server_negotiation.cc:407] Connection header received
0928 06:33:45.575537 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:45.575539 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:45.575552 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:45.575606 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:45.576168 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.576684 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.577946 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.578156 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.578245 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:45.580740 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:45.580742 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:45.580742 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:45.580750 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:45.582437 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:45.583020 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:45.583021 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:45.583021 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:45.583087 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:45.583369 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:45.583370 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:45.583371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:45.583526 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:45.583615 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:45.584247 (+ 632us) server_negotiation.cc:299] Negotiation successful
0928 06:33:45.584313 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 06:33:45.585007 23854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:45.574755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40365 (local address 127.0.0.1:48792)
0928 06:33:45.574888 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:45.574929 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:45.575430 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:45.575623 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:45.575627 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:45.575879 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:45.576067 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:45.576071 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.576791 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.576793 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:45.577855 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:45.577857 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.578277 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.578279 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:45.578423 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:45.578974 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:45.578984 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:45.580643 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:45.582529 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:45.582535 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:45.582537 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:45.582915 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:45.583165 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:45.583166 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:45.583166 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:45.583311 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:45.583954 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:45.583960 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:45.584113 (+ 153us) client_negotiation.cc:769] Sending connection context
0928 06:33:45.584220 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 06:33:45.584367 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":273,"thread_start_us":74,"threads_started":1}
I20260928 06:33:45.585762 23851 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.157.125" port: 40377 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:33:45.585999 23853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:45.574621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58878 (local address 127.16.157.124:44455)
0928 06:33:45.574972 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:45.574976 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:45.575008 (+ 32us) server_negotiation.cc:407] Connection header received
0928 06:33:45.575450 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:45.575463 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:45.575489 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:45.576269 (+ 780us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:45.576319 (+ 50us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.577493 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.578430 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.578826 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.578893 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:45.581430 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:45.581438 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:45.581439 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:45.581446 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:45.583520 (+ 2074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:45.584750 (+ 1230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:45.584753 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:45.584753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:45.584813 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:45.585088 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:45.585090 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:45.585090 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:45.585234 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:45.585397 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:45.585850 (+ 453us) server_negotiation.cc:299] Negotiation successful
0928 06:33:45.585927 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":78,"threads_started":1}
Sep 28 06:33:45 dist-test-slave-rrj5 krb5kdc[23073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577225, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.157.124@KRBTEST.COM for oryx/127.16.157.125@KRBTEST.COM
I20260928 06:33:45.592463 23854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:45.585997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.125:40377 (local address 127.0.0.1:33896)
0928 06:33:45.586352 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:45.586363 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:45.586413 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:45.586555 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:45.586555 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:45.586688 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:45.586720 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:45.586722 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.587357 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.587358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:45.588070 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:45.588073 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.588416 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.588417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:45.588472 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:45.588808 (+ 336us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:45.588812 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:45.590039 (+ 1227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:45.591606 (+ 1567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:45.591607 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:45.591607 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:45.591763 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:45.591966 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:45.591967 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:45.591967 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:45.592016 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:45.592262 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:45.592262 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:45.592294 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 06:33:45.592343 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 06:33:45.592408 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341}
I20260928 06:33:45.592532 23780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:45.586090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33896 (local address 127.16.157.125:40377)
0928 06:33:45.586145 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:45.586147 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:45.586423 (+ 276us) server_negotiation.cc:407] Connection header received
0928 06:33:45.586457 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:45.586458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:45.586469 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:45.586538 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:45.586783 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.587259 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.588151 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.588347 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.588396 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:45.590120 (+ 1724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:45.590122 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:45.590123 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:45.590133 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:45.591525 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:45.591834 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:45.591835 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:45.591835 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:45.591903 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:45.592075 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:45.592076 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:45.592076 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:45.592203 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:45.592266 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:45.592367 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 06:33:45.592454 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 06:33:45.592898 23851 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.157.124" port: 44455 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:33:45 dist-test-slave-rrj5 krb5kdc[23073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577225, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.157.124@KRBTEST.COM for oryx/127.16.157.124@KRBTEST.COM
I20260928 06:33:45.599006 23853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:45.593187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58882 (local address 127.16.157.124:44455)
0928 06:33:45.593218 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:45.593220 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:45.593228 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:33:45.593284 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:45.593285 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:45.593294 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:45.593341 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:45.593601 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.594051 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.594681 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.594895 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.594986 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:45.596577 (+ 1591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:45.596578 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:45.596578 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:45.596584 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:45.597969 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:45.598312 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:45.598313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:45.598313 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:45.598386 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:45.598580 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:45.598581 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:45.598581 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:45.598667 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:45.598713 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:45.598850 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 06:33:45.598944 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 06:33:45.599103 23854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:45.593125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.124:44455 (local address 127.0.0.1:58882)
0928 06:33:45.593173 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:45.593184 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:45.593227 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:45.593342 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:45.593342 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:45.593465 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:45.593493 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:45.593494 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.594137 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.594138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:45.594600 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:45.594602 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.594979 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.594980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:45.595025 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:45.595408 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:45.595412 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:45.596484 (+ 1072us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:45.598067 (+ 1583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:45.598068 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:45.598069 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:45.598228 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:45.598447 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:45.598448 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:45.598448 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:45.598516 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:45.598726 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:45.598726 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:45.598764 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 06:33:45.598836 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 06:33:45.599005 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":59}
I20260928 06:33:45.600642 23851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61915ec9e9cd43f8b944b77b8ba364f1: Bootstrap starting.
I20260928 06:33:45.601603 23851 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61915ec9e9cd43f8b944b77b8ba364f1: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:45.601903 23851 log.cc:824] T 00000000000000000000000000000000 P 61915ec9e9cd43f8b944b77b8ba364f1: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:45.602677 23851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61915ec9e9cd43f8b944b77b8ba364f1: No bootstrap required, opened a new log
I20260928 06:33:45.604537 23851 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61915ec9e9cd43f8b944b77b8ba364f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a9edcc4ed794902a9d229837466efd2" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40365 } } peers { permanent_uuid: "9728b0a476874f8d84efd61d0b15a7eb" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 40377 } } peers { permanent_uuid: "61915ec9e9cd43f8b944b77b8ba364f1" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 44455 } }
I20260928 06:33:45.604684 23851 raft_consensus.cc:374] T 00000000000000000000000000000000 P 61915ec9e9cd43f8b944b77b8ba364f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:45.604715 23851 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61915ec9e9cd43f8b944b77b8ba364f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61915ec9e9cd43f8b944b77b8ba364f1, State: Initialized, Role: FOLLOWER
I20260928 06:33:45.604831 23851 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61915ec9e9cd43f8b944b77b8ba364f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a9edcc4ed794902a9d229837466efd2" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40365 } } peers { permanent_uuid: "9728b0a476874f8d84efd61d0b15a7eb" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 40377 } } peers { permanent_uuid: "61915ec9e9cd43f8b944b77b8ba364f1" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 44455 } }
I20260928 06:33:45.605136 23851 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61915ec9e9cd43f8b944b77b8ba364f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:33:45.605111 23856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61915ec9e9cd43f8b944b77b8ba364f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a9edcc4ed794902a9d229837466efd2" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40365 } } peers { permanent_uuid: "9728b0a476874f8d84efd61d0b15a7eb" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 40377 } } peers { permanent_uuid: "61915ec9e9cd43f8b944b77b8ba364f1" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 44455 } } }
I20260928 06:33:45.605208 23856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61915ec9e9cd43f8b944b77b8ba364f1 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 06:33:45.606973 23867 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 61915ec9e9cd43f8b944b77b8ba364f1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:33:45.607072 23867 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:33:45.608999 23783 master_runner.cc:428] Detected that this master 61915ec9e9cd43f8b944b77b8ba364f1 is joining an existing cluster
I20260928 06:33:45.609068 23783 master_runner.cc:432] Initiating AddMaster RPC to add 127.16.157.124:44455
I20260928 06:33:45.617333 23853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:45.610665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58892 (local address 127.16.157.124:44455)
0928 06:33:45.610700 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:45.610702 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:45.610708 (+ 6us) server_negotiation.cc:407] Connection header received
0928 06:33:45.610723 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:45.610723 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:45.610732 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:45.610779 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:45.611136 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.611848 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.612954 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.613216 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.613253 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:45.614524 (+ 1271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:45.614525 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:45.614526 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:45.614532 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:45.616175 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:45.616586 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:45.616587 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:45.616587 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:45.616637 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:45.616822 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:45.616822 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:45.616823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:45.616941 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:45.616981 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:45.617174 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 06:33:45.617242 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:33:45.617672 23874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:45.610347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.124:44455 (local address 127.0.0.1:58892)
0928 06:33:45.610452 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:45.610461 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:45.610515 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:45.610811 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:45.610812 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:45.611012 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:45.611062 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:45.611064 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.611937 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.611937 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:45.612863 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:45.612865 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.613338 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.613339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:45.613393 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:45.613929 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:45.613935 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:45.614430 (+ 495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:45.616283 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:45.616284 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:45.616284 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:45.616509 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:45.616704 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:45.616705 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:45.616706 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:45.616758 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:45.617021 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:45.617022 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:45.617071 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 06:33:45.617446 (+ 375us) client_negotiation.cc:240] Negotiation successful
0928 06:33:45.617549 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"mutex_wait_us":40,"thread_start_us":70,"threads_started":1}
I20260928 06:33:45.618772 23159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:45.610159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48794 (local address 127.16.157.126:40365)
0928 06:33:45.610482 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:45.610485 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:45.610494 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:33:45.610517 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:45.610518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:45.610528 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:45.610579 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:45.611120 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.611850 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.612961 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.613265 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.613304 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:45.614933 (+ 1629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:45.614934 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:45.614934 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:45.614942 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:45.616888 (+ 1946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:45.617273 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:45.617274 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:45.617274 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:45.617327 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:45.618242 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:45.618243 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:45.618243 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:45.618385 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:45.618435 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:45.618614 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 06:33:45.618686 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299}
I20260928 06:33:45.618785 23873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:45.610113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40365 (local address 127.0.0.1:48794)
0928 06:33:45.610279 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:45.610290 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:45.610338 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:45.610675 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:45.610676 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:45.610914 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:45.611018 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:45.611019 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.611937 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.611937 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:45.612863 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:45.612865 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.613374 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.613375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:45.613429 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:45.614065 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:45.614070 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:45.614824 (+ 754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:45.616986 (+ 2162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:45.616987 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:45.616987 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:45.617192 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:45.617642 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:45.617642 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:45.617643 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:45.617992 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:45.618476 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:45.618476 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:45.618508 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 06:33:45.618583 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 06:33:45.618681 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":2,"thread_start_us":93,"threads_started":1}
Sep 28 06:33:45 dist-test-slave-rrj5 krb5kdc[23073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577224, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.157.126@KRBTEST.COM for oryx/127.16.157.124@KRBTEST.COM
I20260928 06:33:45.637387 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 23182
I20260928 06:33:45.644574 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.65:36871
--local_ip_for_outbound_sockets=127.16.157.65
--tserver_master_addrs=127.16.157.126:40365
--webserver_port=35469
--webserver_interface=127.16.157.65
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/krb5kdc/oryx_127.16.157.65.keytab
--principal=oryx/127.16.157.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:39253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.16.157.126:40365,127.16.157.125:40377
--tserver_master_addrs=127.16.157.126:40365,127.16.157.125:40377,127.16.157.124:44455 with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:33:45.649127 23681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:45.621034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.124:44455 (local address 127.0.0.1:58900)
0928 06:33:45.621071 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:45.621082 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:45.621121 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:45.621292 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:45.621293 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:45.621445 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:45.621497 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:45.621498 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.622327 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.622328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:45.623058 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:45.623061 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.623498 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.623499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:45.623566 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:45.623975 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:45.623980 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:45.625210 (+ 1230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:45.647933 (+ 22723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:45.647934 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:45.647935 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:45.648197 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:45.648430 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:45.648431 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:45.648431 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:45.648488 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:45.648773 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:45.648774 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:45.648825 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 06:33:45.648889 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 06:33:45.649021 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260928 06:33:45.649466 23853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:45.621130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58900 (local address 127.16.157.124:44455)
0928 06:33:45.621162 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:45.621164 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:45.621171 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:33:45.621199 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:45.621200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:45.621208 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:45.621252 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:45.621601 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.622250 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.623166 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.623424 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.623496 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:45.625298 (+ 1802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:45.625299 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:45.625300 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:45.625307 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:45.647660 (+ 22353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:45.648302 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:45.648303 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:45.648304 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:45.648362 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:45.648552 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:45.648552 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:45.648553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:45.648690 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:45.648746 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:45.649302 (+ 556us) server_negotiation.cc:299] Negotiation successful
0928 06:33:45.649380 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 06:33:45.649950 23118 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.16.157.124:44455: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "61915ec9e9cd43f8b944b77b8ba364f1"
member_type: NON_VOTER
last_known_addr {
host: "127.16.157.124"
port: 44455
}
attrs {
promote: true
}
}
dest_uuid: "3a9edcc4ed794902a9d229837466efd2"
cas_config_opid_index: 5
I20260928 06:33:45.650177 23118 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a9edcc4ed794902a9d229837466efd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3a9edcc4ed794902a9d229837466efd2" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40365 } } peers { permanent_uuid: "9728b0a476874f8d84efd61d0b15a7eb" member_type: NON_VOTER last_known_addr { host: "127.16.157.125" port: 40377 } attrs { promote: true } } peers { permanent_uuid: "61915ec9e9cd43f8b944b77b8ba364f1" member_type: NON_VOTER last_known_addr { host: "127.16.157.124" port: 44455 } attrs { promote: true } }
I20260928 06:33:45.650550 23683 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3a9edcc4ed794902a9d229837466efd2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 61915ec9e9cd43f8b944b77b8ba364f1 (127.16.157.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3a9edcc4ed794902a9d229837466efd2" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40365 } } peers { permanent_uuid: "9728b0a476874f8d84efd61d0b15a7eb" member_type: NON_VOTER last_known_addr { host: "127.16.157.125" port: 40377 } attrs { promote: true } } peers { permanent_uuid: "61915ec9e9cd43f8b944b77b8ba364f1" member_type: NON_VOTER last_known_addr { host: "127.16.157.124" port: 44455 } attrs { promote: true } } }
I20260928 06:33:45.651506 23683 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.16.157.124:44455
I20260928 06:33:45.651599 23691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a9edcc4ed794902a9d229837466efd2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "3a9edcc4ed794902a9d229837466efd2" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3a9edcc4ed794902a9d229837466efd2" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40365 } } peers { permanent_uuid: "9728b0a476874f8d84efd61d0b15a7eb" member_type: NON_VOTER last_known_addr { host: "127.16.157.125" port: 40377 } attrs { promote: true } } peers { permanent_uuid: "61915ec9e9cd43f8b944b77b8ba364f1" member_type: NON_VOTER last_known_addr { host: "127.16.157.124" port: 44455 } attrs { promote: true } } }
I20260928 06:33:45.651724 23691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a9edcc4ed794902a9d229837466efd2 [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:45.652241 23825 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61915ec9e9cd43f8b944b77b8ba364f1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:45.653148 23825 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 61915ec9e9cd43f8b944b77b8ba364f1 [term 1 FOLLOWER]: Refusing update from remote peer 3a9edcc4ed794902a9d229837466efd2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 06:33:45.653416 23691 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3a9edcc4ed794902a9d229837466efd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61915ec9e9cd43f8b944b77b8ba364f1" member_type: NON_VOTER last_known_addr { host: "127.16.157.124" port: 44455 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:33:45.653542 23783 master.cc:537] Master@127.16.157.124:44455 shutting down...
W20260928 06:33:45.654078 23103 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260928 06:33:45.656852 23783 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 61915ec9e9cd43f8b944b77b8ba364f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:33:45.656939 23783 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 61915ec9e9cd43f8b944b77b8ba364f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:33:45.656996 23783 tablet_replica.cc:330] T 00000000000000000000000000000000 P 61915ec9e9cd43f8b944b77b8ba364f1: stopping tablet replica
I20260928 06:33:45.662695 23780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:45.614543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33902 (local address 127.16.157.125:40377)
0928 06:33:45.614587 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:45.614589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:45.619509 (+ 4920us) server_negotiation.cc:407] Connection header received
0928 06:33:45.619617 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:45.619617 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:45.619628 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:45.619657 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:45.620054 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.620785 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.622117 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.622404 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.622442 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:45.659498 (+ 37056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:45.659501 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:45.659501 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:45.659509 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:45.661261 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:45.661673 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:45.661674 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:45.661674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:45.661730 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:45.662109 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:45.662110 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:45.662110 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:45.662258 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:45.662310 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:45.662520 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 06:33:45.662603 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 06:33:45.664417 23873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:45.619228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.125:40377 (local address 127.0.0.1:33902)
0928 06:33:45.619267 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:45.619276 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:45.619537 (+ 261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:45.619720 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:45.619721 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:45.619878 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:45.619927 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:45.619929 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.620996 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.620997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:45.622028 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:45.622030 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.622514 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.622514 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:45.622577 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:45.658095 (+ 35518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:45.659016 (+ 921us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:45.659382 (+ 366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:45.661363 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:45.661363 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:45.661364 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:45.661585 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:45.661956 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:45.661956 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:45.661957 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:45.662031 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:45.662362 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:45.662363 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:45.662407 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 06:33:45.664209 (+ 1802us) client_negotiation.cc:240] Negotiation successful
0928 06:33:45.664323 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 06:33:45.665246 23681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:45.651716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.125:40377 (local address 127.0.0.1:33912)
0928 06:33:45.654841 (+ 3125us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:45.654853 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:45.654900 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:45.661813 (+ 6913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:45.661814 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:45.661816 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:33:45.661865 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:45.661867 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.662831 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.662832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:45.664123 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:45.664126 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.664968 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.664969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:45.665057 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:45.665077 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 06:33:45.665138 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 06:33:45.665178 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3111,"thread_start_us":80,"threads_started":1}
I20260928 06:33:45.665343 23877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:45.651909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33912 (local address 127.16.157.125:40377)
0928 06:33:45.661651 (+ 9742us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:45.661654 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:45.661663 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:33:45.661688 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:45.661689 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:45.661708 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:45.661784 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:33:45.661925 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.662710 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.664215 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.664874 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.664970 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:45.665044 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:45.665224 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 06:33:45.665268 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9719,"thread_start_us":74,"threads_started":1}
I20260928 06:33:45.665562 23681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:45.665397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.124:44455 (local address 127.0.0.1:58916)
0928 06:33:45.665502 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:45.665539 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.124:44455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260928 06:33:45.665604 23738 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb [term 1 LEARNER]: Refusing update from remote peer 3a9edcc4ed794902a9d229837466efd2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20260928 06:33:45.665676 23103 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3a9edcc4ed794902a9d229837466efd2 -> Peer 61915ec9e9cd43f8b944b77b8ba364f1 (127.16.157.124:44455): Couldn't send request to peer 61915ec9e9cd43f8b944b77b8ba364f1. Status: Network error: Client connection negotiation failed: client connection to 127.16.157.124:44455: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 06:33:45.665846 23691 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3a9edcc4ed794902a9d229837466efd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9728b0a476874f8d84efd61d0b15a7eb" member_type: NON_VOTER last_known_addr { host: "127.16.157.125" port: 40377 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 06:33:45.666281 23738 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 61915ec9e9cd43f8b944b77b8ba364f1 (127.16.157.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3a9edcc4ed794902a9d229837466efd2" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40365 } } peers { permanent_uuid: "9728b0a476874f8d84efd61d0b15a7eb" member_type: NON_VOTER last_known_addr { host: "127.16.157.125" port: 40377 } attrs { promote: true } } peers { permanent_uuid: "61915ec9e9cd43f8b944b77b8ba364f1" member_type: NON_VOTER last_known_addr { host: "127.16.157.124" port: 44455 } attrs { promote: true } } }
I20260928 06:33:45.667336 23768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a9edcc4ed794902a9d229837466efd2. Latest consensus state: current_term: 1 leader_uuid: "3a9edcc4ed794902a9d229837466efd2" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3a9edcc4ed794902a9d229837466efd2" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40365 } } peers { permanent_uuid: "9728b0a476874f8d84efd61d0b15a7eb" member_type: NON_VOTER last_known_addr { host: "127.16.157.125" port: 40377 } attrs { promote: true } } peers { permanent_uuid: "61915ec9e9cd43f8b944b77b8ba364f1" member_type: NON_VOTER last_known_addr { host: "127.16.157.124" port: 44455 } attrs { promote: true } } }
I20260928 06:33:45.667430 23768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb [sys.catalog]: This master's current role is: LEARNER
I20260928 06:33:45.667466 23768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3a9edcc4ed794902a9d229837466efd2" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3a9edcc4ed794902a9d229837466efd2" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40365 } } peers { permanent_uuid: "9728b0a476874f8d84efd61d0b15a7eb" member_type: NON_VOTER last_known_addr { host: "127.16.157.125" port: 40377 } attrs { promote: true } } peers { permanent_uuid: "61915ec9e9cd43f8b944b77b8ba364f1" member_type: NON_VOTER last_known_addr { host: "127.16.157.124" port: 44455 } attrs { promote: true } } }
I20260928 06:33:45.667518 23768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb [sys.catalog]: This master's current role is: LEARNER
I20260928 06:33:45.667722 23691 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 3a9edcc4ed794902a9d229837466efd2: attempting to promote NON_VOTER 9728b0a476874f8d84efd61d0b15a7eb to VOTER
I20260928 06:33:45.667933 23691 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a9edcc4ed794902a9d229837466efd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3a9edcc4ed794902a9d229837466efd2" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40365 } } peers { permanent_uuid: "9728b0a476874f8d84efd61d0b15a7eb" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 40377 } attrs { promote: false } } peers { permanent_uuid: "61915ec9e9cd43f8b944b77b8ba364f1" member_type: NON_VOTER last_known_addr { host: "127.16.157.124" port: 44455 } attrs { promote: true } }
I20260928 06:33:45.668493 23681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:45.668328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.124:44455 (local address 127.0.0.1:58930)
0928 06:33:45.668445 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:45.668468 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.124:44455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260928 06:33:45.668653 23738 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb [term 1 LEARNER]: Refusing update from remote peer 3a9edcc4ed794902a9d229837466efd2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 06:33:45.668870 23681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:45.668738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.124:44455 (local address 127.0.0.1:58944)
0928 06:33:45.668825 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:45.668848 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.124:44455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260928 06:33:45.668941 23683 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3a9edcc4ed794902a9d229837466efd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9728b0a476874f8d84efd61d0b15a7eb" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 40377 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260928 06:33:45.670080 23783 master.cc:552] Master@127.16.157.124:44455 shutdown complete.
I20260928 06:33:45.670130 23783 master_runner.cc:305] Clearing existing system tablet
I20260928 06:33:45.670259 23783 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:45.670305 23783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20260928 06:33:45.671041 23103 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3a9edcc4ed794902a9d229837466efd2 -> Peer 61915ec9e9cd43f8b944b77b8ba364f1 (127.16.157.124:44455): Couldn't send request to peer 61915ec9e9cd43f8b944b77b8ba364f1. Status: Network error: Client connection negotiation failed: client connection to 127.16.157.124:44455: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 06:33:45.671144 23783 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:33:45.671634 23882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:45.671834 23683 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3a9edcc4ed794902a9d229837466efd2 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9728b0a476874f8d84efd61d0b15a7eb (127.16.157.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3a9edcc4ed794902a9d229837466efd2" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40365 } } peers { permanent_uuid: "9728b0a476874f8d84efd61d0b15a7eb" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 40377 } attrs { promote: false } } peers { permanent_uuid: "61915ec9e9cd43f8b944b77b8ba364f1" member_type: NON_VOTER last_known_addr { host: "127.16.157.124" port: 44455 } attrs { promote: true } } }
I20260928 06:33:45.671984 23738 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9728b0a476874f8d84efd61d0b15a7eb (127.16.157.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3a9edcc4ed794902a9d229837466efd2" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40365 } } peers { permanent_uuid: "9728b0a476874f8d84efd61d0b15a7eb" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 40377 } attrs { promote: false } } peers { permanent_uuid: "61915ec9e9cd43f8b944b77b8ba364f1" member_type: NON_VOTER last_known_addr { host: "127.16.157.124" port: 44455 } attrs { promote: true } } }
I20260928 06:33:45.672665 23683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a9edcc4ed794902a9d229837466efd2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "3a9edcc4ed794902a9d229837466efd2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3a9edcc4ed794902a9d229837466efd2" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40365 } } peers { permanent_uuid: "9728b0a476874f8d84efd61d0b15a7eb" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 40377 } attrs { promote: false } } peers { permanent_uuid: "61915ec9e9cd43f8b944b77b8ba364f1" member_type: NON_VOTER last_known_addr { host: "127.16.157.124" port: 44455 } attrs { promote: true } } }
I20260928 06:33:45.672753 23683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a9edcc4ed794902a9d229837466efd2 [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:45.672804 23683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a9edcc4ed794902a9d229837466efd2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3a9edcc4ed794902a9d229837466efd2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3a9edcc4ed794902a9d229837466efd2" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40365 } } peers { permanent_uuid: "9728b0a476874f8d84efd61d0b15a7eb" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 40377 } attrs { promote: false } } peers { permanent_uuid: "61915ec9e9cd43f8b944b77b8ba364f1" member_type: NON_VOTER last_known_addr { host: "127.16.157.124" port: 44455 } attrs { promote: true } } }
I20260928 06:33:45.672844 23683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a9edcc4ed794902a9d229837466efd2 [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:45.672998 23768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3a9edcc4ed794902a9d229837466efd2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3a9edcc4ed794902a9d229837466efd2" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40365 } } peers { permanent_uuid: "9728b0a476874f8d84efd61d0b15a7eb" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 40377 } attrs { promote: false } } peers { permanent_uuid: "61915ec9e9cd43f8b944b77b8ba364f1" member_type: NON_VOTER last_known_addr { host: "127.16.157.124" port: 44455 } attrs { promote: true } } }
I20260928 06:33:45.673071 23768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:33:45.673223 23783 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:45.673281 23783 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s
I20260928 06:33:45.673318 23783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/master-2/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/master-2/wal
uuid: "61915ec9e9cd43f8b944b77b8ba364f1"
format_stamp: "Formatted at 2026-09-28 06:33:45 on dist-test-slave-rrj5"
server_key: "c6465dca8fb68e2088ec45ced64cab34"
server_key_iv: "2110e4274a754df2344634cca4ac03de"
server_key_version: "encryptionkey@0"
I20260928 06:33:45.674079 23783 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 61915ec9e9cd43f8b944b77b8ba364f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:33:45.675433 23783 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 61915ec9e9cd43f8b944b77b8ba364f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 06:33:45.675504 23783 log.cc:1196] T 00000000000000000000000000000000 P 61915ec9e9cd43f8b944b77b8ba364f1: Deleting WAL directory at /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 06:33:45.675704 23783 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 61915ec9e9cd43f8b944b77b8ba364f1: Deleting consensus metadata
I20260928 06:33:45.675836 23783 master_runner.cc:315] Copying system tablet from 127.16.157.126:40365
I20260928 06:33:45.676434 23783 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 61915ec9e9cd43f8b944b77b8ba364f1: tablet copy: Beginning tablet copy session from remote peer at address 127.16.157.126:40365
I20260928 06:33:45.684396 23159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:45.676738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48796 (local address 127.16.157.126:40365)
0928 06:33:45.676777 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:45.676779 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:45.677009 (+ 230us) server_negotiation.cc:407] Connection header received
0928 06:33:45.677104 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:45.677105 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:45.677114 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:45.677144 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:45.677564 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.678394 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.679527 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.679826 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.679865 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:45.681105 (+ 1240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:45.681107 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:45.681107 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:45.681114 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:45.683084 (+ 1970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:45.683475 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:45.683476 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:45.683476 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:45.683537 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:45.683770 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:45.683771 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:45.683771 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:45.683916 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:45.683962 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:45.684226 (+ 264us) server_negotiation.cc:299] Negotiation successful
0928 06:33:45.684303 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:33:45.684746 23888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:45.676823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40365 (local address 127.0.0.1:48796)
0928 06:33:45.676941 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:45.676950 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:45.677035 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:45.677203 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:45.677204 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:45.677406 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:45.677476 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:45.677478 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.678541 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.678542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:45.679422 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:45.679425 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.679941 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.679942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:45.680023 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:45.680519 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:45.680525 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:45.681002 (+ 477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:45.683180 (+ 2178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:45.683181 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:45.683181 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:45.683395 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:45.683610 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:45.683611 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:45.683611 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:45.683700 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:45.684032 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:45.684033 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:45.684078 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 06:33:45.684515 (+ 437us) client_negotiation.cc:240] Negotiation successful
0928 06:33:45.684623 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":58,"threads_started":1}
I20260928 06:33:45.684974 23138 tablet_copy_service.cc:140] P 3a9edcc4ed794902a9d229837466efd2: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 61915ec9e9cd43f8b944b77b8ba364f1 ({username='oryx', principal='oryx/127.16.157.124@KRBTEST.COM'} at 127.0.0.1:48796)
I20260928 06:33:45.685047 23138 tablet_copy_service.cc:161] P 3a9edcc4ed794902a9d229837466efd2: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 61915ec9e9cd43f8b944b77b8ba364f1 at {username='oryx', principal='oryx/127.16.157.124@KRBTEST.COM'} at 127.0.0.1:48796: session id = 61915ec9e9cd43f8b944b77b8ba364f1-00000000000000000000000000000000
I20260928 06:33:45.685611 23138 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 3a9edcc4ed794902a9d229837466efd2: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 06:33:45.686070 23783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 06:33:45.687357 23783 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 61915ec9e9cd43f8b944b77b8ba364f1: tablet copy: Starting download of 0 data blocks...
I20260928 06:33:45.687496 23783 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 61915ec9e9cd43f8b944b77b8ba364f1: tablet copy: Starting download of 1 WAL segments...
I20260928 06:33:45.688169 23783 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 61915ec9e9cd43f8b944b77b8ba364f1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 06:33:45.688974 23138 tablet_copy_service.cc:342] P 3a9edcc4ed794902a9d229837466efd2: Request end of tablet copy session 61915ec9e9cd43f8b944b77b8ba364f1-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.16.157.124@KRBTEST.COM'} at 127.0.0.1:48796
I20260928 06:33:45.689018 23138 tablet_copy_service.cc:433] P 3a9edcc4ed794902a9d229837466efd2: ending tablet copy session 61915ec9e9cd43f8b944b77b8ba364f1-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 61915ec9e9cd43f8b944b77b8ba364f1
I20260928 06:33:45.691942 23783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:45.693527 23892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:45.693883 23783 server_base.cc:1034] running on GCE node
W20260928 06:33:45.698122 23889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:45.698252 23890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:45.698463 23783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:45.698601 23783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:45.699741 23783 hybrid_clock.cc:648] HybridClock initialized: now 1790577225699714 us; error 45 us; skew 500 ppm
Sep 28 06:33:45 dist-test-slave-rrj5 krb5kdc[23073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577225, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.157.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:45.701313 23783 init.cc:377] Logged in from keytab as oryx/127.16.157.124@KRBTEST.COM (short username oryx)
I20260928 06:33:45.701602 23783 webserver.cc:460] Webserver started at http://127.16.157.124:44527/ using document root <none> and password file <none>
I20260928 06:33:45.701714 23783 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:45.701741 23783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:45.702534 23783 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:33:45.703117 23899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:45.703246 23783 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:45.703279 23783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/master-2/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/master-2/wal
uuid: "61915ec9e9cd43f8b944b77b8ba364f1"
format_stamp: "Formatted at 2026-09-28 06:33:45 on dist-test-slave-rrj5"
server_key: "c6465dca8fb68e2088ec45ced64cab34"
server_key_iv: "2110e4274a754df2344634cca4ac03de"
server_key_version: "encryptionkey@0"
I20260928 06:33:45.703327 23783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:45.725374 23783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:45.727353 23783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:45.730400 23783 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.124:44455
I20260928 06:33:45.730782 23783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/master-2/data/info.pb
I20260928 06:33:45.731928 23951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.124:44455 every 8 connection(s)
I20260928 06:33:45.732772 23952 sys_catalog.cc:263] Verifying existing consensus state
I20260928 06:33:45.733343 23952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61915ec9e9cd43f8b944b77b8ba364f1: Bootstrap starting.
I20260928 06:33:45.736485 23952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61915ec9e9cd43f8b944b77b8ba364f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:33:45.736920 23952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61915ec9e9cd43f8b944b77b8ba364f1: Bootstrap complete.
I20260928 06:33:45.737228 23952 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61915ec9e9cd43f8b944b77b8ba364f1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3a9edcc4ed794902a9d229837466efd2" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40365 } } peers { permanent_uuid: "9728b0a476874f8d84efd61d0b15a7eb" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 40377 } attrs { promote: false } } peers { permanent_uuid: "61915ec9e9cd43f8b944b77b8ba364f1" member_type: NON_VOTER last_known_addr { host: "127.16.157.124" port: 44455 } attrs { promote: true } }
I20260928 06:33:45.737421 23952 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61915ec9e9cd43f8b944b77b8ba364f1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 61915ec9e9cd43f8b944b77b8ba364f1, State: Initialized, Role: LEARNER
I20260928 06:33:45.737509 23952 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61915ec9e9cd43f8b944b77b8ba364f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3a9edcc4ed794902a9d229837466efd2" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40365 } } peers { permanent_uuid: "9728b0a476874f8d84efd61d0b15a7eb" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 40377 } attrs { promote: false } } peers { permanent_uuid: "61915ec9e9cd43f8b944b77b8ba364f1" member_type: NON_VOTER last_known_addr { host: "127.16.157.124" port: 44455 } attrs { promote: true } }
I20260928 06:33:45.737846 23952 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61915ec9e9cd43f8b944b77b8ba364f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:33:45.737807 23955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61915ec9e9cd43f8b944b77b8ba364f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3a9edcc4ed794902a9d229837466efd2" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40365 } } peers { permanent_uuid: "9728b0a476874f8d84efd61d0b15a7eb" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 40377 } attrs { promote: false } } peers { permanent_uuid: "61915ec9e9cd43f8b944b77b8ba364f1" member_type: NON_VOTER last_known_addr { host: "127.16.157.124" port: 44455 } attrs { promote: true } } }
I20260928 06:33:45.737977 23955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61915ec9e9cd43f8b944b77b8ba364f1 [sys.catalog]: This master's current role is: LEARNER
I20260928 06:33:45.740607 23966 catalog_manager.cc:1259] Loaded cluster ID: fc73c8eec1414312b3fe0164efd101a8
I20260928 06:33:45.740701 23966 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 61915ec9e9cd43f8b944b77b8ba364f1: loading cluster ID for follower catalog manager: success
I20260928 06:33:45.741880 23966 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 61915ec9e9cd43f8b944b77b8ba364f1: acquiring CA information for follower catalog manager: success
I20260928 06:33:45.742295 23966 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 61915ec9e9cd43f8b944b77b8ba364f1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 06:33:45.757342 23875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:45.757565 23875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:45.757620 23875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:45.758787 23875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:45.758868 23875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:45.758916 23875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.65
I20260928 06:33:45.760425 23875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:39253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/krb5kdc/oryx_127.16.157.65.keytab
--principal=oryx/127.16.157.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.65:36871
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.157.65
--webserver_port=35469
--tserver_master_addrs=127.16.157.126:40365,127.16.157.125:40377,127.16.157.124:44455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.65
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:45.760701 23875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:45.760929 23875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:45.765975 23971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:45.766288 23972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:45.766482 23974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:45.766830 23875 server_base.cc:1034] running on GCE node
I20260928 06:33:45.767038 23875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:45.767342 23875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:45.768514 23875 hybrid_clock.cc:648] HybridClock initialized: now 1790577225768494 us; error 43 us; skew 500 ppm
Sep 28 06:33:45 dist-test-slave-rrj5 krb5kdc[23073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577225, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.157.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:45.771180 23875 init.cc:377] Logged in from keytab as oryx/127.16.157.65@KRBTEST.COM (short username oryx)
I20260928 06:33:45.771692 23875 webserver.cc:460] Webserver started at http://127.16.157.65:35469/ using document root <none> and password file <none>
I20260928 06:33:45.771942 23875 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:45.772025 23875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:45.773386 23875 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:33:45.774067 23981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:45.774236 23875 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:45.774322 23875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-0/wal
uuid: "d8d60b2a738a49288570af1432d7c119"
format_stamp: "Formatted at 2026-09-28 06:33:44 on dist-test-slave-rrj5"
server_key: "bb0a23ec650544e64ab1d00707f008cb"
server_key_iv: "60d321c371f770cf654fd6c051a0f00e"
server_key_version: "encryptionkey@0"
I20260928 06:33:45.774684 23875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:45.819123 23875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:45.821242 23875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:45.821412 23875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:45.821708 23875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:33:45.822026 23875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:33:45.822088 23875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:45.822130 23875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:33:45.822156 23875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:45.828825 23875 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.65:36871
I20260928 06:33:45.828899 24095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.65:36871 every 8 connection(s)
I20260928 06:33:45.829237 23875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
Sep 28 06:33:45 dist-test-slave-rrj5 krb5kdc[23073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577225, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.157.65@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM
Sep 28 06:33:45 dist-test-slave-rrj5 krb5kdc[23073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577225, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.157.65@KRBTEST.COM for oryx/127.16.157.124@KRBTEST.COM
I20260928 06:33:45.838387 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 23875
I20260928 06:33:45.838563 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 23317
Sep 28 06:33:45 dist-test-slave-rrj5 krb5kdc[23073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577225, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.157.65@KRBTEST.COM for oryx/127.16.157.125@KRBTEST.COM
I20260928 06:33:45.843657 23159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:45.830460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:41513 (local address 127.16.157.126:40365)
0928 06:33:45.830520 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:45.830523 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:45.831154 (+ 631us) server_negotiation.cc:407] Connection header received
0928 06:33:45.831250 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:45.831251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:45.831265 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:45.831303 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:45.832412 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.832910 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.834506 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.834811 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.834855 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:45.839795 (+ 4940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:45.839797 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:45.839797 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:45.839806 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:45.841993 (+ 2187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:45.842483 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:45.842484 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:45.842484 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:45.842557 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:45.842856 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:45.842857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:45.842858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:45.842998 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:45.843045 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:45.843475 (+ 430us) server_negotiation.cc:299] Negotiation successful
0928 06:33:45.843560 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 06:33:45.843658 23877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:45.830307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:38725 (local address 127.16.157.125:40377)
0928 06:33:45.830349 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:45.830351 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:45.830362 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:33:45.831039 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:45.831041 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:45.831053 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:45.831113 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:45.832177 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.833009 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.834506 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.834811 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.834863 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:45.840114 (+ 5251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:45.840115 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:45.840116 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:45.840123 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:45.841894 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:45.842483 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:45.842484 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:45.842484 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:45.842557 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:45.842869 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:45.842870 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:45.842870 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:45.843009 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:45.843050 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:45.843472 (+ 422us) server_negotiation.cc:299] Negotiation successful
0928 06:33:45.843564 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:33:45.844230 24100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:45.830027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.125:40377 (local address 127.16.157.65:38725)
0928 06:33:45.830206 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:45.830251 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:45.830914 (+ 663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:45.831206 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:45.831211 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:45.831793 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:45.832059 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:45.832065 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.833088 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.833088 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:45.834396 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:45.834399 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.834952 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.834953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:45.835114 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:45.836159 (+ 1045us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:45.836167 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:45.840014 (+ 3847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:45.841991 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:45.841999 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:45.842001 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:45.842389 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:45.842628 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:45.842629 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:45.842629 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:45.842774 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:45.843097 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:45.843105 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:45.843291 (+ 186us) client_negotiation.cc:769] Sending connection context
0928 06:33:45.843792 (+ 501us) client_negotiation.cc:240] Negotiation successful
0928 06:33:45.844007 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":65,"thread_start_us":100,"threads_started":1}
I20260928 06:33:45.844563 24102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:45.830302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40365 (local address 127.16.157.65:41513)
0928 06:33:45.831095 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:45.831104 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:45.831179 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:45.831387 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:45.831387 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:45.832273 (+ 886us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:45.832319 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:45.832321 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.833022 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.833023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:45.834396 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:45.834399 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.834955 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.834955 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:45.835114 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:45.836235 (+ 1121us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:45.836239 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:45.839727 (+ 3488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:45.842072 (+ 2345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:45.842072 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:45.842073 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:45.842389 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:45.842633 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:45.842633 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:45.842634 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:45.842774 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:45.843097 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:45.843105 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:45.843291 (+ 186us) client_negotiation.cc:769] Sending connection context
0928 06:33:45.844120 (+ 829us) client_negotiation.cc:240] Negotiation successful
0928 06:33:45.844204 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":774,"mutex_wait_us":37,"thread_start_us":58,"threads_started":1}
I20260928 06:33:45.844774 24098 heartbeater.cc:347] Connected to a master server at 127.16.157.125:40377
I20260928 06:33:45.844972 24098 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:45.845170 24098 heartbeater.cc:511] Master 127.16.157.125:40377 requested a full tablet report, sending...
I20260928 06:33:45.845749 23728 ts_manager.cc:194] Registered new tserver with Master: d8d60b2a738a49288570af1432d7c119 (127.16.157.65:36871)
I20260928 06:33:45.846323 24097 heartbeater.cc:347] Connected to a master server at 127.16.157.126:40365
I20260928 06:33:45.846381 24097 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:45.846455 24097 heartbeater.cc:511] Master 127.16.157.126:40365 requested a full tablet report, sending...
I20260928 06:33:45.846793 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.66:39405
--local_ip_for_outbound_sockets=127.16.157.66
--tserver_master_addrs=127.16.157.126:40365
--webserver_port=36103
--webserver_interface=127.16.157.66
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/krb5kdc/oryx_127.16.157.66.keytab
--principal=oryx/127.16.157.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:39253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.16.157.126:40365,127.16.157.125:40377
--tserver_master_addrs=127.16.157.126:40365,127.16.157.125:40377,127.16.157.124:44455 with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:33:45.848398 23118 ts_manager.cc:194] Re-registered known tserver with Master: d8d60b2a738a49288570af1432d7c119 (127.16.157.65:36871)
I20260928 06:33:45.848901 23118 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.157.65@KRBTEST.COM'} at 127.16.157.65:41513
I20260928 06:33:45.849326 24101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:45.830028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.124:44455 (local address 127.16.157.65:41803)
0928 06:33:45.830420 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:45.830429 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:45.830925 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:45.831296 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:45.831297 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:45.831768 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:45.832060 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:45.832065 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.834284 (+ 2219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.834285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:45.835147 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:45.835150 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.835698 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.835699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:45.835744 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:45.836400 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:45.836406 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:45.839459 (+ 3053us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:45.847935 (+ 8476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:45.847936 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:45.847936 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:45.848115 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:45.848374 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:45.848375 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:45.848375 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:45.848438 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:45.848717 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:45.848718 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:45.848756 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 06:33:45.848799 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 06:33:45.848904 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"mutex_wait_us":5,"thread_start_us":85,"threads_started":1}
I20260928 06:33:45.849692 24103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:45.830534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:41803 (local address 127.16.157.124:44455)
0928 06:33:45.830689 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:45.830691 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:45.830701 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:33:45.831179 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:45.831180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:45.831192 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:45.831441 (+ 249us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:45.833129 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.834181 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.835255 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:45.835615 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:45.835691 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:45.846162 (+ 10471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:45.846164 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:45.846164 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:45.846171 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:45.847821 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:45.848202 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:45.848203 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:45.848204 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:45.848291 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:45.848516 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:45.848516 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:45.848517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:45.848644 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:45.849440 (+ 796us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:45.849483 (+ 43us) server_negotiation.cc:299] Negotiation successful
0928 06:33:45.849596 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":77,"threads_started":1}
I20260928 06:33:45.849997 24096 heartbeater.cc:347] Connected to a master server at 127.16.157.124:44455
I20260928 06:33:45.850075 24096 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:45.850150 24096 heartbeater.cc:511] Master 127.16.157.124:44455 requested a full tablet report, sending...
I20260928 06:33:45.850499 23907 ts_manager.cc:194] Registered new tserver with Master: d8d60b2a738a49288570af1432d7c119 (127.16.157.65:36871)
W20260928 06:33:45.943967 24104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:45.944161 24104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:45.944185 24104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:45.945346 24104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:45.945391 24104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:45.945417 24104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.66
I20260928 06:33:45.946916 24104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:39253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/krb5kdc/oryx_127.16.157.66.keytab
--principal=oryx/127.16.157.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.66:39405
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.157.66
--webserver_port=36103
--tserver_master_addrs=127.16.157.126:40365,127.16.157.125:40377,127.16.157.124:44455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.66
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:45.947177 24104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:45.947431 24104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:45.953364 24109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:45.953387 24113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:45.953410 24111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:45.953711 24104 server_base.cc:1034] running on GCE node
I20260928 06:33:45.953922 24104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:45.954166 24104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:45.955327 24104 hybrid_clock.cc:648] HybridClock initialized: now 1790577225955304 us; error 38 us; skew 500 ppm
Sep 28 06:33:45 dist-test-slave-rrj5 krb5kdc[23073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577225, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.157.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:45.957928 24104 init.cc:377] Logged in from keytab as oryx/127.16.157.66@KRBTEST.COM (short username oryx)
I20260928 06:33:45.958421 24104 webserver.cc:460] Webserver started at http://127.16.157.66:36103/ using document root <none> and password file <none>
I20260928 06:33:45.958689 24104 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:45.958772 24104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:45.960146 24104 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:45.960776 24120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:45.961023 24104 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:45.961119 24104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-1/wal
uuid: "ae0e85a2c6794239a27fd60294c0a03f"
format_stamp: "Formatted at 2026-09-28 06:33:44 on dist-test-slave-rrj5"
server_key: "93b8c96a036421cbc2418082ce5c31be"
server_key_iv: "039579f7e28b97bf97275b8946e85a08"
server_key_version: "encryptionkey@0"
I20260928 06:33:45.961521 24104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:45.994297 24104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:45.996899 24104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:45.997081 24104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:45.997416 24104 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:33:45.997833 24104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:33:45.997884 24104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:45.997918 24104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:33:45.997947 24104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:46.005697 24104 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.66:39405
I20260928 06:33:46.005734 24234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.66:39405 every 8 connection(s)
I20260928 06:33:46.006156 24104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
I20260928 06:33:46.013432 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 24104
I20260928 06:33:46.013687 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 23452
Sep 28 06:33:46 dist-test-slave-rrj5 krb5kdc[23073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577225, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.157.66@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM
Sep 28 06:33:46 dist-test-slave-rrj5 krb5kdc[23073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577225, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.157.66@KRBTEST.COM for oryx/127.16.157.124@KRBTEST.COM
I20260928 06:33:46.019541 23159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:46.006991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.66:44665 (local address 127.16.157.126:40365)
0928 06:33:46.007050 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:46.007053 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:46.007270 (+ 217us) server_negotiation.cc:407] Connection header received
0928 06:33:46.008304 (+ 1034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:46.008306 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:46.008318 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:46.008349 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:46.009330 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:46.010088 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:46.011761 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:46.012101 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:46.012152 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:46.015733 (+ 3581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:46.015734 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:46.015735 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:46.015743 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:46.017862 (+ 2119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:46.018383 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:46.018385 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:46.018385 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:46.018446 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:46.018717 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:46.018718 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:46.018718 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:46.018876 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:46.018919 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:46.019299 (+ 380us) server_negotiation.cc:299] Negotiation successful
0928 06:33:46.019388 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 06:33:46.019804 24103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:46.007398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.66:36105 (local address 127.16.157.124:44455)
0928 06:33:46.007439 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:46.007441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:46.007450 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:33:46.008141 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:46.008143 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:46.008159 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:46.008213 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:46.009317 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:46.010120 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:46.011779 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:46.012101 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:46.012140 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:46.016037 (+ 3897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:46.016038 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:46.016039 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:46.016047 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:46.018570 (+ 2523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:46.018936 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:46.018937 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:46.018937 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:46.018995 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:46.019200 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:46.019201 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:46.019201 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:46.019342 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:46.019388 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:46.019610 (+ 222us) server_negotiation.cc:299] Negotiation successful
0928 06:33:46.019708 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:33:46.020401 24239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:46.006945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40365 (local address 127.16.157.66:44665)
0928 06:33:46.007126 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:46.007180 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:46.008196 (+ 1016us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:46.008433 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:46.008434 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:46.008878 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:46.009200 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:46.009207 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:46.010234 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:46.010235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:46.011650 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:46.011653 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:46.012241 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:46.012242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:46.012341 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:46.013096 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:46.013107 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:46.015613 (+ 2506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:46.017954 (+ 2341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:46.017961 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:46.017963 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:46.018284 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:46.018515 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:46.018516 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:46.018516 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:46.018649 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:46.018979 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:46.018985 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:46.019147 (+ 162us) client_negotiation.cc:769] Sending connection context
0928 06:33:46.019749 (+ 602us) client_negotiation.cc:240] Negotiation successful
0928 06:33:46.020090 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":152,"thread_start_us":93,"threads_started":1}
I20260928 06:33:46.020406 24240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:46.007230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.124:44455 (local address 127.16.157.66:36105)
0928 06:33:46.007382 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:46.007393 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:46.008019 (+ 626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:46.008297 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:46.008301 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:46.008917 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:46.009200 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:46.009206 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:46.010249 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:46.010250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:46.011651 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:46.011654 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:46.012212 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:46.012213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:46.012339 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:46.013131 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:46.013137 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:46.015933 (+ 2796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:46.018664 (+ 2731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:46.018665 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:46.018665 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:46.018849 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:46.019069 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:46.019070 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:46.019070 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:46.019129 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:46.019441 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:46.019442 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:46.019485 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 06:33:46.019982 (+ 497us) client_negotiation.cc:240] Negotiation successful
0928 06:33:46.020179 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":243,"thread_start_us":65,"threads_started":1}
I20260928 06:33:46.021081 24236 heartbeater.cc:347] Connected to a master server at 127.16.157.126:40365
I20260928 06:33:46.021091 24235 heartbeater.cc:347] Connected to a master server at 127.16.157.124:44455
I20260928 06:33:46.021190 24235 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:46.021189 24236 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:46.021395 24235 heartbeater.cc:511] Master 127.16.157.124:44455 requested a full tablet report, sending...
I20260928 06:33:46.021395 24236 heartbeater.cc:511] Master 127.16.157.126:40365 requested a full tablet report, sending...
I20260928 06:33:46.021893 23118 ts_manager.cc:194] Re-registered known tserver with Master: ae0e85a2c6794239a27fd60294c0a03f (127.16.157.66:39405)
I20260928 06:33:46.021963 23907 ts_manager.cc:194] Registered new tserver with Master: ae0e85a2c6794239a27fd60294c0a03f (127.16.157.66:39405)
I20260928 06:33:46.022331 23118 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.157.66@KRBTEST.COM'} at 127.16.157.66:44665
I20260928 06:33:46.022696 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.67:33283
--local_ip_for_outbound_sockets=127.16.157.67
--tserver_master_addrs=127.16.157.126:40365
--webserver_port=46135
--webserver_interface=127.16.157.67
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/krb5kdc/oryx_127.16.157.67.keytab
--principal=oryx/127.16.157.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:39253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.16.157.126:40365,127.16.157.125:40377
--tserver_master_addrs=127.16.157.126:40365,127.16.157.125:40377,127.16.157.124:44455 with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 06:33:46 dist-test-slave-rrj5 krb5kdc[23073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577225, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.157.66@KRBTEST.COM for oryx/127.16.157.125@KRBTEST.COM
I20260928 06:33:46.028443 23877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:46.007208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.66:52723 (local address 127.16.157.125:40377)
0928 06:33:46.007321 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:46.007323 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:46.007414 (+ 91us) server_negotiation.cc:407] Connection header received
0928 06:33:46.008271 (+ 857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:46.008273 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:46.008284 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:46.008331 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:46.009315 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:46.010130 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:46.011766 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:46.012062 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:46.012103 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:46.025608 (+ 13505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:46.025609 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:46.025609 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:46.025617 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:46.027300 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:46.027642 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:46.027643 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:46.027643 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:46.027696 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:46.027903 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:46.027904 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:46.027904 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:46.028031 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:46.028103 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:46.028271 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 06:33:46.028354 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 06:33:46.028463 24241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:46.006945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.125:40377 (local address 127.16.157.66:52723)
0928 06:33:46.007346 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:46.007358 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:46.008047 (+ 689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:46.008380 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:46.008381 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:46.008878 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:46.009200 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:46.009206 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:46.010253 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:46.010253 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:46.011655 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:46.011657 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:46.012197 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:46.012198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:46.012339 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:46.013186 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:46.013189 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:46.025481 (+ 12292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:46.027380 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:46.027381 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:46.027381 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:46.027553 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:46.027772 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:46.027773 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:46.027773 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:46.027830 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:46.028107 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:46.028108 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:46.028149 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 06:33:46.028238 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 06:33:46.028352 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"mutex_wait_us":103,"thread_start_us":90,"threads_started":1}
I20260928 06:33:46.029049 24237 heartbeater.cc:347] Connected to a master server at 127.16.157.125:40377
I20260928 06:33:46.029102 24237 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:46.029183 24237 heartbeater.cc:511] Master 127.16.157.125:40377 requested a full tablet report, sending...
I20260928 06:33:46.029474 23728 ts_manager.cc:194] Registered new tserver with Master: ae0e85a2c6794239a27fd60294c0a03f (127.16.157.66:39405)
W20260928 06:33:46.122205 24242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:46.122422 24242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:46.122463 24242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:46.123710 24242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:46.123770 24242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:46.123814 24242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.67
I20260928 06:33:46.125504 24242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:39253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/krb5kdc/oryx_127.16.157.67.keytab
--principal=oryx/127.16.157.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.67:33283
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.157.67
--webserver_port=46135
--tserver_master_addrs=127.16.157.126:40365,127.16.157.125:40377,127.16.157.124:44455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.67
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:46.125769 24242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:46.126008 24242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:46.131076 24248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:46.131037 24247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:46.131026 24250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:46.131368 24242 server_base.cc:1034] running on GCE node
I20260928 06:33:46.131603 24242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:46.131850 24242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:46.133028 24242 hybrid_clock.cc:648] HybridClock initialized: now 1790577226133011 us; error 35 us; skew 500 ppm
Sep 28 06:33:46 dist-test-slave-rrj5 krb5kdc[23073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577226, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.157.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:46.136096 24242 init.cc:377] Logged in from keytab as oryx/127.16.157.67@KRBTEST.COM (short username oryx)
I20260928 06:33:46.136718 24242 webserver.cc:460] Webserver started at http://127.16.157.67:46135/ using document root <none> and password file <none>
I20260928 06:33:46.137006 24242 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:46.137104 24242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:46.139045 24242 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:46.139842 24257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:46.140031 24242 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:46.140127 24242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-2/wal
uuid: "6cec1f1c72de4f28a819e930d1f014b5"
format_stamp: "Formatted at 2026-09-28 06:33:45 on dist-test-slave-rrj5"
server_key: "5a170e70375f49610fc6f368d8496ef1"
server_key_iv: "93c71114e136000ff0853b81f32e9e21"
server_key_version: "encryptionkey@0"
I20260928 06:33:46.140503 24242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:46.148114 24242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:46.150343 24242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:46.150501 24242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:46.150785 24242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:33:46.151067 24242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:33:46.151119 24242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:46.151160 24242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:33:46.151185 24242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:46.157727 24242 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.67:33283
I20260928 06:33:46.157770 24371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.67:33283 every 8 connection(s)
I20260928 06:33:46.158077 24242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
Sep 28 06:33:46 dist-test-slave-rrj5 krb5kdc[23073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577226, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.157.67@KRBTEST.COM for oryx/127.16.157.124@KRBTEST.COM
Sep 28 06:33:46 dist-test-slave-rrj5 krb5kdc[23073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577226, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.157.67@KRBTEST.COM for oryx/127.16.157.125@KRBTEST.COM
Sep 28 06:33:46 dist-test-slave-rrj5 krb5kdc[23073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577226, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.157.67@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM
I20260928 06:33:46.168128 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 24242
I20260928 06:33:46.171420 24103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:46.159310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:51747 (local address 127.16.157.124:44455)
0928 06:33:46.159380 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:46.159382 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:46.159400 (+ 18us) server_negotiation.cc:407] Connection header received
0928 06:33:46.159820 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:46.159821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:46.159833 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:46.159866 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:46.160936 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:46.161714 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:46.163231 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:46.163538 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:46.163574 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:46.167083 (+ 3509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:46.167084 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:46.167084 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:46.167091 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:46.169687 (+ 2596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:46.170475 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:46.170476 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:46.170477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:46.170524 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:46.170724 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:46.170724 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:46.170725 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:46.170846 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:46.170902 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:46.171238 (+ 336us) server_negotiation.cc:299] Negotiation successful
0928 06:33:46.171322 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:33:46.171540 24378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:46.158810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.125:40377 (local address 127.16.157.67:37775)
0928 06:33:46.159126 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:46.159137 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:46.159717 (+ 580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:46.160051 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:46.160054 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:46.160590 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:46.160823 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:46.160829 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:46.161865 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:46.161866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:46.163136 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:46.163139 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:46.163691 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:46.163692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:46.163799 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:46.164568 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:46.164572 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:46.167315 (+ 2743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:46.169690 (+ 2375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:46.169697 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:46.169699 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:46.170067 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:46.170437 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:46.170438 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:46.170438 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:46.170575 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:46.170873 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:46.170892 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:46.171046 (+ 154us) client_negotiation.cc:769] Sending connection context
0928 06:33:46.171153 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 06:33:46.171322 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":115,"thread_start_us":113,"threads_started":1}
I20260928 06:33:46.171754 24377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:46.158733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.124:44455 (local address 127.16.157.67:51747)
0928 06:33:46.158993 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:46.159034 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:46.159700 (+ 666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:46.160050 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:46.160054 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:46.160542 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:46.160824 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:46.160829 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:46.161852 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:46.161852 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:46.163127 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:46.163130 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:46.163656 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:46.163658 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:46.163797 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:46.164441 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:46.164446 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:46.167001 (+ 2555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:46.169765 (+ 2764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:46.169766 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:46.169767 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:46.170213 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:46.170600 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:46.170601 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:46.170601 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:46.170656 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:46.170925 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:46.170926 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:46.171046 (+ 120us) client_negotiation.cc:769] Sending connection context
0928 06:33:46.171153 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 06:33:46.171639 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":486,"thread_start_us":99,"threads_started":1}
I20260928 06:33:46.171967 23877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:46.159185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:37775 (local address 127.16.157.125:40377)
0928 06:33:46.159378 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:46.159380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:46.159400 (+ 20us) server_negotiation.cc:407] Connection header received
0928 06:33:46.159835 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:46.159836 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:46.159849 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:46.159885 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:46.160946 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:46.161754 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:46.163256 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:46.163548 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:46.163601 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:46.167410 (+ 3809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:46.167411 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:46.167411 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:46.167417 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:46.169538 (+ 2121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:46.170293 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:46.170294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:46.170294 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:46.170360 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:46.170650 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:46.170651 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:46.170651 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:46.170786 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:46.170846 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:46.171768 (+ 922us) server_negotiation.cc:299] Negotiation successful
0928 06:33:46.171847 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169}
I20260928 06:33:46.172716 24372 heartbeater.cc:347] Connected to a master server at 127.16.157.124:44455
I20260928 06:33:46.172824 24372 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:46.173027 24372 heartbeater.cc:511] Master 127.16.157.124:44455 requested a full tablet report, sending...
I20260928 06:33:46.173132 23159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:46.159250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:33321 (local address 127.16.157.126:40365)
0928 06:33:46.159301 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:46.159303 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:46.159537 (+ 234us) server_negotiation.cc:407] Connection header received
0928 06:33:46.159820 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:46.159821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:46.159834 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:46.159866 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:46.160942 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:46.161750 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:46.163231 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:46.163523 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:46.163566 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:46.167389 (+ 3823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:46.167390 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:46.167391 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:46.167398 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:46.169754 (+ 2356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:46.172263 (+ 2509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:46.172264 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:46.172264 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:46.172325 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:46.172567 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:46.172568 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:46.172568 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:46.172718 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:46.172762 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:46.172974 (+ 212us) server_negotiation.cc:299] Negotiation successful
0928 06:33:46.173052 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 06:33:46.173273 24374 heartbeater.cc:347] Connected to a master server at 127.16.157.125:40377
I20260928 06:33:46.173312 24374 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:46.173373 24374 heartbeater.cc:511] Master 127.16.157.125:40377 requested a full tablet report, sending...
I20260928 06:33:46.173447 24376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:46.159019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40365 (local address 127.16.157.67:33321)
0928 06:33:46.159456 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:46.159467 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:46.159709 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:46.160050 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:46.160054 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:46.160541 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:46.160824 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:46.160829 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:46.161852 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:46.161852 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:46.163131 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:46.163133 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:46.163658 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:46.163658 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:46.163797 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:46.164385 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:46.164394 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:46.167001 (+ 2607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:46.171958 (+ 4957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:46.171958 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:46.171959 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:46.172176 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:46.172421 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:46.172422 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:46.172423 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:46.172486 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:46.172812 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:46.172813 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:46.172853 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 06:33:46.173230 (+ 377us) client_negotiation.cc:240] Negotiation successful
0928 06:33:46.173335 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"mutex_wait_us":83,"thread_start_us":61,"threads_started":1}
I20260928 06:33:46.174008 23907 ts_manager.cc:194] Registered new tserver with Master: 6cec1f1c72de4f28a819e930d1f014b5 (127.16.157.67:33283)
I20260928 06:33:46.174333 23728 ts_manager.cc:194] Registered new tserver with Master: 6cec1f1c72de4f28a819e930d1f014b5 (127.16.157.67:33283)
I20260928 06:33:46.174748 24373 heartbeater.cc:347] Connected to a master server at 127.16.157.126:40365
I20260928 06:33:46.174801 24373 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:46.174861 24373 heartbeater.cc:511] Master 127.16.157.126:40365 requested a full tablet report, sending...
I20260928 06:33:46.175069 23118 ts_manager.cc:194] Re-registered known tserver with Master: 6cec1f1c72de4f28a819e930d1f014b5 (127.16.157.67:33283)
I20260928 06:33:46.175513 23118 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.157.67@KRBTEST.COM'} at 127.16.157.67:33321
I20260928 06:33:46.180655 24384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:46.170130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33914 (local address 127.16.157.125:40377)
0928 06:33:46.170233 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:46.170235 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:46.170480 (+ 245us) server_negotiation.cc:407] Connection header received
0928 06:33:46.170505 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:46.170505 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:46.170517 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:46.170590 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:46.171449 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:46.173131 (+ 1682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:46.174881 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:46.175298 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:46.175349 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:46.177092 (+ 1743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:46.177093 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:46.177094 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:46.177101 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:46.179046 (+ 1945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:46.179603 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:46.179604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:46.179605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:46.179662 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:46.179870 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:46.179871 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:46.179871 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:46.180019 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:46.180117 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:46.180455 (+ 338us) server_negotiation.cc:299] Negotiation successful
0928 06:33:46.180541 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":50,"threads_started":1}
I20260928 06:33:46.183823 24103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:46.170493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58950 (local address 127.16.157.124:44455)
0928 06:33:46.171496 (+ 1003us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:46.171498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:46.176259 (+ 4761us) server_negotiation.cc:407] Connection header received
0928 06:33:46.176340 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:46.176341 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:46.176349 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:46.176375 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:46.177040 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:46.177509 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:46.178552 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:46.178856 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:46.178931 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:46.180326 (+ 1395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:46.180328 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:46.180328 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:46.180334 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:46.182556 (+ 2222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:46.183053 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:46.183054 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:46.183054 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:46.183123 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:46.183321 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:46.183322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:46.183322 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:46.183450 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:46.183551 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:46.183652 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 06:33:46.183724 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":991,"thread_start_us":58,"threads_started":1}
I20260928 06:33:46.184432 23159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:46.170088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48800 (local address 127.16.157.126:40365)
0928 06:33:46.173211 (+ 3123us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:46.173213 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:46.173707 (+ 494us) server_negotiation.cc:407] Connection header received
0928 06:33:46.173802 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:46.173803 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:46.173813 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:46.173842 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:46.174593 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:46.175235 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:46.176213 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:46.176527 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:46.176567 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:46.179510 (+ 2943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:46.179511 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:46.179512 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:46.179520 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:46.181796 (+ 2276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:46.183235 (+ 1439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:46.183236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:46.183236 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:46.183295 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:46.183527 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:46.183528 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:46.183529 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:46.183686 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:46.183770 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:46.184296 (+ 526us) server_negotiation.cc:299] Negotiation successful
0928 06:33:46.184368 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3107,"thread_start_us":61,"threads_started":1}
I20260928 06:33:46.185988 23118 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48800:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:33:46.187460 23118 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:33:46.195375 24395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:46.191143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.66:39405 (local address 127.0.0.1:54086)
0928 06:33:46.191417 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:46.191425 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:46.191459 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:46.191759 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:46.191759 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:46.191761 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:33:46.191806 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:46.191808 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:46.192991 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:46.192992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:46.194109 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:46.194112 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:46.195111 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:46.195112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:46.195201 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:46.195205 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:33:46.195255 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 06:33:46.195295 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":5,"thread_start_us":75,"threads_started":1}
I20260928 06:33:46.195472 24400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:46.191149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35872 (local address 127.16.157.67:33283)
0928 06:33:46.191337 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:46.191342 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:46.191378 (+ 36us) server_negotiation.cc:407] Connection header received
0928 06:33:46.191434 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:46.191437 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:46.191456 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:46.191559 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:33:46.191683 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:46.192772 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:46.194209 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:46.194924 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:46.194996 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:46.195073 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:46.195340 (+ 267us) server_negotiation.cc:299] Negotiation successful
0928 06:33:46.195382 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":106,"threads_started":1}
I20260928 06:33:46.195521 24399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:46.190942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.67:33283 (local address 127.0.0.1:35872)
0928 06:33:46.191313 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:46.191324 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:46.191361 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:46.191552 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:46.191553 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:46.191554 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:33:46.191600 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:46.191601 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:46.192864 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:46.192865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:46.194110 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:46.194112 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:46.195114 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:46.195115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:46.195201 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:46.195204 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:33:46.195255 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 06:33:46.195298 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"mutex_wait_us":63,"thread_start_us":89,"threads_started":1}
I20260928 06:33:46.195807 24401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:46.191376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54086 (local address 127.16.157.66:39405)
0928 06:33:46.191549 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:46.191554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:46.191564 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:33:46.191603 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:46.191605 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:46.191622 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:46.191697 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:33:46.191882 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:46.192910 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:46.194208 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:46.194924 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:46.194996 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:46.195079 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:46.195675 (+ 596us) server_negotiation.cc:299] Negotiation successful
0928 06:33:46.195719 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":91,"threads_started":1}
I20260928 06:33:46.196201 24306 tablet_service.cc:1467] Processing CreateTablet for tablet 22bbc5baf6854d27b5320c5a35829061 (DEFAULT_TABLE table=test-table [id=44486d75691043ea8b01d85672671c9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:33:46.196585 24306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22bbc5baf6854d27b5320c5a35829061. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:46.196518 24168 tablet_service.cc:1467] Processing CreateTablet for tablet 22bbc5baf6854d27b5320c5a35829061 (DEFAULT_TABLE table=test-table [id=44486d75691043ea8b01d85672671c9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:33:46.196838 24168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22bbc5baf6854d27b5320c5a35829061. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:46.197629 24396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:46.190924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.65:36871 (local address 127.0.0.1:54214)
0928 06:33:46.191066 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:46.191078 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:46.191324 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:46.192072 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:46.192073 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:46.192076 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:33:46.192120 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:46.192122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:46.193848 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:46.193849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:46.194964 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:46.194967 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:46.197326 (+ 2359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:46.197327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:46.197398 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:46.197403 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:33:46.197440 (+ 37us) client_negotiation.cc:240] Negotiation successful
0928 06:33:46.197491 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":103,"threads_started":1}
I20260928 06:33:46.198369 24397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:46.191019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54214 (local address 127.16.157.65:36871)
0928 06:33:46.191884 (+ 865us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:46.191888 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:46.191899 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:33:46.191939 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:46.191941 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:46.191956 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:46.192199 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:33:46.192276 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:46.193375 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:46.195081 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:46.197182 (+ 2101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:46.197985 (+ 803us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:46.198084 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:46.198117 (+ 33us) server_negotiation.cc:299] Negotiation successful
0928 06:33:46.198171 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":824,"thread_start_us":80,"threads_started":1}
I20260928 06:33:46.199280 24029 tablet_service.cc:1467] Processing CreateTablet for tablet 22bbc5baf6854d27b5320c5a35829061 (DEFAULT_TABLE table=test-table [id=44486d75691043ea8b01d85672671c9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:33:46.199615 24029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22bbc5baf6854d27b5320c5a35829061. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:46.200470 24403 tablet_bootstrap.cc:492] T 22bbc5baf6854d27b5320c5a35829061 P 6cec1f1c72de4f28a819e930d1f014b5: Bootstrap starting.
I20260928 06:33:46.201184 24403 tablet_bootstrap.cc:654] T 22bbc5baf6854d27b5320c5a35829061 P 6cec1f1c72de4f28a819e930d1f014b5: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:46.201550 24403 log.cc:824] T 22bbc5baf6854d27b5320c5a35829061 P 6cec1f1c72de4f28a819e930d1f014b5: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:46.201867 24402 tablet_bootstrap.cc:492] T 22bbc5baf6854d27b5320c5a35829061 P ae0e85a2c6794239a27fd60294c0a03f: Bootstrap starting.
I20260928 06:33:46.202838 24402 tablet_bootstrap.cc:654] T 22bbc5baf6854d27b5320c5a35829061 P ae0e85a2c6794239a27fd60294c0a03f: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:46.203202 24402 log.cc:824] T 22bbc5baf6854d27b5320c5a35829061 P ae0e85a2c6794239a27fd60294c0a03f: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:46.203547 24405 tablet_bootstrap.cc:492] T 22bbc5baf6854d27b5320c5a35829061 P d8d60b2a738a49288570af1432d7c119: Bootstrap starting.
I20260928 06:33:46.204119 24403 tablet_bootstrap.cc:492] T 22bbc5baf6854d27b5320c5a35829061 P 6cec1f1c72de4f28a819e930d1f014b5: No bootstrap required, opened a new log
I20260928 06:33:46.204155 24405 tablet_bootstrap.cc:654] T 22bbc5baf6854d27b5320c5a35829061 P d8d60b2a738a49288570af1432d7c119: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:46.204196 24403 ts_tablet_manager.cc:1397] T 22bbc5baf6854d27b5320c5a35829061 P 6cec1f1c72de4f28a819e930d1f014b5: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260928 06:33:46.204501 24405 log.cc:824] T 22bbc5baf6854d27b5320c5a35829061 P d8d60b2a738a49288570af1432d7c119: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:46.205215 24405 tablet_bootstrap.cc:492] T 22bbc5baf6854d27b5320c5a35829061 P d8d60b2a738a49288570af1432d7c119: No bootstrap required, opened a new log
I20260928 06:33:46.205309 24405 ts_tablet_manager.cc:1397] T 22bbc5baf6854d27b5320c5a35829061 P d8d60b2a738a49288570af1432d7c119: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:33:46.207391 24405 raft_consensus.cc:348] T 22bbc5baf6854d27b5320c5a35829061 P d8d60b2a738a49288570af1432d7c119 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d60b2a738a49288570af1432d7c119" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 36871 } } peers { permanent_uuid: "6cec1f1c72de4f28a819e930d1f014b5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 33283 } } peers { permanent_uuid: "ae0e85a2c6794239a27fd60294c0a03f" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 39405 } }
I20260928 06:33:46.207542 24405 raft_consensus.cc:374] T 22bbc5baf6854d27b5320c5a35829061 P d8d60b2a738a49288570af1432d7c119 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:46.207587 24405 raft_consensus.cc:729] T 22bbc5baf6854d27b5320c5a35829061 P d8d60b2a738a49288570af1432d7c119 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8d60b2a738a49288570af1432d7c119, State: Initialized, Role: FOLLOWER
I20260928 06:33:46.207552 24403 raft_consensus.cc:348] T 22bbc5baf6854d27b5320c5a35829061 P 6cec1f1c72de4f28a819e930d1f014b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d60b2a738a49288570af1432d7c119" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 36871 } } peers { permanent_uuid: "6cec1f1c72de4f28a819e930d1f014b5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 33283 } } peers { permanent_uuid: "ae0e85a2c6794239a27fd60294c0a03f" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 39405 } }
I20260928 06:33:46.207737 24403 raft_consensus.cc:374] T 22bbc5baf6854d27b5320c5a35829061 P 6cec1f1c72de4f28a819e930d1f014b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:46.207695 24405 consensus_queue.cc:260] T 22bbc5baf6854d27b5320c5a35829061 P d8d60b2a738a49288570af1432d7c119 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d60b2a738a49288570af1432d7c119" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 36871 } } peers { permanent_uuid: "6cec1f1c72de4f28a819e930d1f014b5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 33283 } } peers { permanent_uuid: "ae0e85a2c6794239a27fd60294c0a03f" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 39405 } }
I20260928 06:33:46.207787 24403 raft_consensus.cc:729] T 22bbc5baf6854d27b5320c5a35829061 P 6cec1f1c72de4f28a819e930d1f014b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cec1f1c72de4f28a819e930d1f014b5, State: Initialized, Role: FOLLOWER
I20260928 06:33:46.207971 24405 ts_tablet_manager.cc:1428] T 22bbc5baf6854d27b5320c5a35829061 P d8d60b2a738a49288570af1432d7c119: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:33:46.207933 24403 consensus_queue.cc:260] T 22bbc5baf6854d27b5320c5a35829061 P 6cec1f1c72de4f28a819e930d1f014b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d60b2a738a49288570af1432d7c119" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 36871 } } peers { permanent_uuid: "6cec1f1c72de4f28a819e930d1f014b5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 33283 } } peers { permanent_uuid: "ae0e85a2c6794239a27fd60294c0a03f" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 39405 } }
I20260928 06:33:46.208246 24097 heartbeater.cc:503] Master 127.16.157.126:40365 was elected leader, sending a full tablet report...
I20260928 06:33:46.209272 24402 tablet_bootstrap.cc:492] T 22bbc5baf6854d27b5320c5a35829061 P ae0e85a2c6794239a27fd60294c0a03f: No bootstrap required, opened a new log
I20260928 06:33:46.209372 24402 ts_tablet_manager.cc:1397] T 22bbc5baf6854d27b5320c5a35829061 P ae0e85a2c6794239a27fd60294c0a03f: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260928 06:33:46.209743 24403 ts_tablet_manager.cc:1428] T 22bbc5baf6854d27b5320c5a35829061 P 6cec1f1c72de4f28a819e930d1f014b5: Time spent starting tablet: real 0.006s user 0.003s sys 0.001s
I20260928 06:33:46.209861 24373 heartbeater.cc:503] Master 127.16.157.126:40365 was elected leader, sending a full tablet report...
I20260928 06:33:46.212543 24402 raft_consensus.cc:348] T 22bbc5baf6854d27b5320c5a35829061 P ae0e85a2c6794239a27fd60294c0a03f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d60b2a738a49288570af1432d7c119" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 36871 } } peers { permanent_uuid: "6cec1f1c72de4f28a819e930d1f014b5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 33283 } } peers { permanent_uuid: "ae0e85a2c6794239a27fd60294c0a03f" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 39405 } }
I20260928 06:33:46.212664 24402 raft_consensus.cc:374] T 22bbc5baf6854d27b5320c5a35829061 P ae0e85a2c6794239a27fd60294c0a03f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:46.212684 24402 raft_consensus.cc:729] T 22bbc5baf6854d27b5320c5a35829061 P ae0e85a2c6794239a27fd60294c0a03f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae0e85a2c6794239a27fd60294c0a03f, State: Initialized, Role: FOLLOWER
I20260928 06:33:46.212812 24402 consensus_queue.cc:260] T 22bbc5baf6854d27b5320c5a35829061 P ae0e85a2c6794239a27fd60294c0a03f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d60b2a738a49288570af1432d7c119" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 36871 } } peers { permanent_uuid: "6cec1f1c72de4f28a819e930d1f014b5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 33283 } } peers { permanent_uuid: "ae0e85a2c6794239a27fd60294c0a03f" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 39405 } }
I20260928 06:33:46.213081 24236 heartbeater.cc:503] Master 127.16.157.126:40365 was elected leader, sending a full tablet report...
I20260928 06:33:46.213075 24402 ts_tablet_manager.cc:1428] T 22bbc5baf6854d27b5320c5a35829061 P ae0e85a2c6794239a27fd60294c0a03f: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260928 06:33:46.221761 24396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:46.218808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.124:44455 (local address 127.0.0.1:58954)
0928 06:33:46.218871 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:46.218885 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:46.218936 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:46.219073 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:46.219074 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:46.219075 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:33:46.219115 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:46.219116 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:46.220003 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:46.220004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:46.220737 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:46.220739 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:46.221500 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:46.221501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:46.221552 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:46.221556 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:33:46.221670 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 06:33:46.221708 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 06:33:46.221783 24103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:46.218860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58954 (local address 127.16.157.124:44455)
0928 06:33:46.218911 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:46.218913 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:46.218923 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:33:46.218990 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:46.218990 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:46.219010 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:46.219072 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:33:46.219156 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:46.219923 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:46.220828 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:46.221411 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:46.221486 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:46.221558 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:46.221677 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 06:33:46.221715 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:33:46.222189 23926 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 61915ec9e9cd43f8b944b77b8ba364f1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260928 06:33:46.223047 23955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61915ec9e9cd43f8b944b77b8ba364f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a9edcc4ed794902a9d229837466efd2. Latest consensus state: current_term: 1 leader_uuid: "3a9edcc4ed794902a9d229837466efd2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3a9edcc4ed794902a9d229837466efd2" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40365 } } peers { permanent_uuid: "9728b0a476874f8d84efd61d0b15a7eb" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 40377 } attrs { promote: false } } peers { permanent_uuid: "61915ec9e9cd43f8b944b77b8ba364f1" member_type: NON_VOTER last_known_addr { host: "127.16.157.124" port: 44455 } attrs { promote: true } } }
I20260928 06:33:46.223135 23955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61915ec9e9cd43f8b944b77b8ba364f1 [sys.catalog]: This master's current role is: LEARNER
W20260928 06:33:46.257870 24238 tablet.cc:2369] T 22bbc5baf6854d27b5320c5a35829061 P ae0e85a2c6794239a27fd60294c0a03f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:33:46.324723 24408 raft_consensus.cc:482] T 22bbc5baf6854d27b5320c5a35829061 P d8d60b2a738a49288570af1432d7c119 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:33:46.324844 24408 raft_consensus.cc:504] T 22bbc5baf6854d27b5320c5a35829061 P d8d60b2a738a49288570af1432d7c119 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d60b2a738a49288570af1432d7c119" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 36871 } } peers { permanent_uuid: "6cec1f1c72de4f28a819e930d1f014b5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 33283 } } peers { permanent_uuid: "ae0e85a2c6794239a27fd60294c0a03f" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 39405 } }
I20260928 06:33:46.325268 24408 leader_election.cc:288] T 22bbc5baf6854d27b5320c5a35829061 P d8d60b2a738a49288570af1432d7c119 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6cec1f1c72de4f28a819e930d1f014b5 (127.16.157.67:33283), ae0e85a2c6794239a27fd60294c0a03f (127.16.157.66:39405)
I20260928 06:33:46.328660 24401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:46.325639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:53269 (local address 127.16.157.66:39405)
0928 06:33:46.325683 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:46.325686 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:46.325696 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:33:46.325787 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:46.325789 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:46.325798 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:46.325844 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:33:46.326046 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:46.326682 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:46.327893 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:46.328296 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:46.328384 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:46.328393 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:46.328545 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 06:33:46.328592 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:33:46.328656 24102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:46.325478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.66:39405 (local address 127.16.157.65:53269)
0928 06:33:46.325655 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:46.325664 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:46.325694 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:46.325889 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:46.325889 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:46.325891 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:33:46.325956 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:46.325958 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:46.326775 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:46.326776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:46.327801 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:46.327803 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:46.328384 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:46.328385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:46.328464 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:46.328468 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:33:46.328525 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 06:33:46.328572 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":2}
I20260928 06:33:46.328835 24400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:46.325512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:42207 (local address 127.16.157.67:33283)
0928 06:33:46.325601 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:46.325603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:46.325629 (+ 26us) server_negotiation.cc:407] Connection header received
0928 06:33:46.325659 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:46.325660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:46.325672 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:46.325721 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:33:46.325901 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:46.326433 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:46.327815 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:46.328363 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:46.328436 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:46.328444 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:46.328693 (+ 249us) server_negotiation.cc:299] Negotiation successful
0928 06:33:46.328746 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 06:33:46.329001 24101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:46.325433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.67:33283 (local address 127.16.157.65:42207)
0928 06:33:46.325499 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:46.325511 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:46.325558 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:46.325746 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:46.325747 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:46.325749 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:33:46.325830 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:46.325832 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:46.326537 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:46.326538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:46.327721 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:46.327723 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:46.328440 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:46.328441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:46.328535 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:46.328537 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:33:46.328583 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 06:33:46.328614 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 06:33:46.329140 24189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22bbc5baf6854d27b5320c5a35829061" candidate_uuid: "d8d60b2a738a49288570af1432d7c119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0e85a2c6794239a27fd60294c0a03f" is_pre_election: true
I20260928 06:33:46.329329 24189 raft_consensus.cc:2454] T 22bbc5baf6854d27b5320c5a35829061 P ae0e85a2c6794239a27fd60294c0a03f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8d60b2a738a49288570af1432d7c119 in term 0.
I20260928 06:33:46.329401 24326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22bbc5baf6854d27b5320c5a35829061" candidate_uuid: "d8d60b2a738a49288570af1432d7c119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cec1f1c72de4f28a819e930d1f014b5" is_pre_election: true
I20260928 06:33:46.329550 24326 raft_consensus.cc:2454] T 22bbc5baf6854d27b5320c5a35829061 P 6cec1f1c72de4f28a819e930d1f014b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8d60b2a738a49288570af1432d7c119 in term 0.
I20260928 06:33:46.329643 23982 leader_election.cc:302] T 22bbc5baf6854d27b5320c5a35829061 P d8d60b2a738a49288570af1432d7c119 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0e85a2c6794239a27fd60294c0a03f, d8d60b2a738a49288570af1432d7c119; no voters:
I20260928 06:33:46.329766 24408 raft_consensus.cc:2781] T 22bbc5baf6854d27b5320c5a35829061 P d8d60b2a738a49288570af1432d7c119 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:33:46.329825 24408 raft_consensus.cc:482] T 22bbc5baf6854d27b5320c5a35829061 P d8d60b2a738a49288570af1432d7c119 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:33:46.329856 24408 raft_consensus.cc:3037] T 22bbc5baf6854d27b5320c5a35829061 P d8d60b2a738a49288570af1432d7c119 [term 0 FOLLOWER]: Advancing to term 1
W20260928 06:33:46.330271 24099 tablet.cc:2369] T 22bbc5baf6854d27b5320c5a35829061 P d8d60b2a738a49288570af1432d7c119: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:33:46.330719 24408 raft_consensus.cc:504] T 22bbc5baf6854d27b5320c5a35829061 P d8d60b2a738a49288570af1432d7c119 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d60b2a738a49288570af1432d7c119" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 36871 } } peers { permanent_uuid: "6cec1f1c72de4f28a819e930d1f014b5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 33283 } } peers { permanent_uuid: "ae0e85a2c6794239a27fd60294c0a03f" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 39405 } }
I20260928 06:33:46.330878 24408 leader_election.cc:288] T 22bbc5baf6854d27b5320c5a35829061 P d8d60b2a738a49288570af1432d7c119 [CANDIDATE]: Term 1 election: Requested vote from peers 6cec1f1c72de4f28a819e930d1f014b5 (127.16.157.67:33283), ae0e85a2c6794239a27fd60294c0a03f (127.16.157.66:39405)
I20260928 06:33:46.331079 24326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22bbc5baf6854d27b5320c5a35829061" candidate_uuid: "d8d60b2a738a49288570af1432d7c119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cec1f1c72de4f28a819e930d1f014b5"
I20260928 06:33:46.331132 24189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22bbc5baf6854d27b5320c5a35829061" candidate_uuid: "d8d60b2a738a49288570af1432d7c119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0e85a2c6794239a27fd60294c0a03f"
I20260928 06:33:46.331185 24326 raft_consensus.cc:3037] T 22bbc5baf6854d27b5320c5a35829061 P 6cec1f1c72de4f28a819e930d1f014b5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:46.331198 24189 raft_consensus.cc:3037] T 22bbc5baf6854d27b5320c5a35829061 P ae0e85a2c6794239a27fd60294c0a03f [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:46.332078 24189 raft_consensus.cc:2454] T 22bbc5baf6854d27b5320c5a35829061 P ae0e85a2c6794239a27fd60294c0a03f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8d60b2a738a49288570af1432d7c119 in term 1.
I20260928 06:33:46.332078 24326 raft_consensus.cc:2454] T 22bbc5baf6854d27b5320c5a35829061 P 6cec1f1c72de4f28a819e930d1f014b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8d60b2a738a49288570af1432d7c119 in term 1.
I20260928 06:33:46.332275 23985 leader_election.cc:302] T 22bbc5baf6854d27b5320c5a35829061 P d8d60b2a738a49288570af1432d7c119 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cec1f1c72de4f28a819e930d1f014b5, d8d60b2a738a49288570af1432d7c119; no voters:
I20260928 06:33:46.332369 24408 raft_consensus.cc:2781] T 22bbc5baf6854d27b5320c5a35829061 P d8d60b2a738a49288570af1432d7c119 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:33:46.332507 24408 raft_consensus.cc:686] T 22bbc5baf6854d27b5320c5a35829061 P d8d60b2a738a49288570af1432d7c119 [term 1 LEADER]: Becoming Leader. State: Replica: d8d60b2a738a49288570af1432d7c119, State: Running, Role: LEADER
I20260928 06:33:46.332615 24408 consensus_queue.cc:237] T 22bbc5baf6854d27b5320c5a35829061 P d8d60b2a738a49288570af1432d7c119 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d60b2a738a49288570af1432d7c119" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 36871 } } peers { permanent_uuid: "6cec1f1c72de4f28a819e930d1f014b5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 33283 } } peers { permanent_uuid: "ae0e85a2c6794239a27fd60294c0a03f" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 39405 } }
I20260928 06:33:46.333415 23118 catalog_manager.cc:5495] T 22bbc5baf6854d27b5320c5a35829061 P d8d60b2a738a49288570af1432d7c119 reported cstate change: term changed from 0 to 1, leader changed from <none> to d8d60b2a738a49288570af1432d7c119 (127.16.157.65). New cstate: current_term: 1 leader_uuid: "d8d60b2a738a49288570af1432d7c119" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d60b2a738a49288570af1432d7c119" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 36871 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6cec1f1c72de4f28a819e930d1f014b5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 33283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae0e85a2c6794239a27fd60294c0a03f" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 39405 } health_report { overall_health: UNKNOWN } } }
I20260928 06:33:46.334021 24390 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a9edcc4ed794902a9d229837466efd2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3a9edcc4ed794902a9d229837466efd2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3a9edcc4ed794902a9d229837466efd2" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40365 } } peers { permanent_uuid: "9728b0a476874f8d84efd61d0b15a7eb" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 40377 } attrs { promote: false } } peers { permanent_uuid: "61915ec9e9cd43f8b944b77b8ba364f1" member_type: NON_VOTER last_known_addr { host: "127.16.157.124" port: 44455 } attrs { promote: true } } }
I20260928 06:33:46.334132 24390 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a9edcc4ed794902a9d229837466efd2 [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:46.334802 24394 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 3a9edcc4ed794902a9d229837466efd2: attempting to promote NON_VOTER 61915ec9e9cd43f8b944b77b8ba364f1 to VOTER
I20260928 06:33:46.334924 24394 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a9edcc4ed794902a9d229837466efd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "3a9edcc4ed794902a9d229837466efd2" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40365 } } peers { permanent_uuid: "9728b0a476874f8d84efd61d0b15a7eb" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 40377 } attrs { promote: false } } peers { permanent_uuid: "61915ec9e9cd43f8b944b77b8ba364f1" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 44455 } attrs { promote: false } }
I20260928 06:33:46.335512 23738 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb [term 1 FOLLOWER]: Refusing update from remote peer 3a9edcc4ed794902a9d229837466efd2: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260928 06:33:46.335537 23926 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 61915ec9e9cd43f8b944b77b8ba364f1 [term 1 LEARNER]: Refusing update from remote peer 3a9edcc4ed794902a9d229837466efd2: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260928 06:33:46.335758 24390 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3a9edcc4ed794902a9d229837466efd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9728b0a476874f8d84efd61d0b15a7eb" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 40377 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260928 06:33:46.335918 24390 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3a9edcc4ed794902a9d229837466efd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61915ec9e9cd43f8b944b77b8ba364f1" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 44455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260928 06:33:46.336325 23691 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3a9edcc4ed794902a9d229837466efd2 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 61915ec9e9cd43f8b944b77b8ba364f1 (127.16.157.124) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "3a9edcc4ed794902a9d229837466efd2" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40365 } } peers { permanent_uuid: "9728b0a476874f8d84efd61d0b15a7eb" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 40377 } attrs { promote: false } } peers { permanent_uuid: "61915ec9e9cd43f8b944b77b8ba364f1" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 44455 } attrs { promote: false } } }
I20260928 06:33:46.337296 23691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a9edcc4ed794902a9d229837466efd2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "3a9edcc4ed794902a9d229837466efd2" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "3a9edcc4ed794902a9d229837466efd2" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40365 } } peers { permanent_uuid: "9728b0a476874f8d84efd61d0b15a7eb" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 40377 } attrs { promote: false } } peers { permanent_uuid: "61915ec9e9cd43f8b944b77b8ba364f1" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 44455 } attrs { promote: false } } }
I20260928 06:33:46.337396 23691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a9edcc4ed794902a9d229837466efd2 [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:46.337463 23691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a9edcc4ed794902a9d229837466efd2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3a9edcc4ed794902a9d229837466efd2" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "3a9edcc4ed794902a9d229837466efd2" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40365 } } peers { permanent_uuid: "9728b0a476874f8d84efd61d0b15a7eb" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 40377 } attrs { promote: false } } peers { permanent_uuid: "61915ec9e9cd43f8b944b77b8ba364f1" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 44455 } attrs { promote: false } } }
I20260928 06:33:46.337531 23691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a9edcc4ed794902a9d229837466efd2 [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:46.337622 23691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a9edcc4ed794902a9d229837466efd2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3a9edcc4ed794902a9d229837466efd2" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "3a9edcc4ed794902a9d229837466efd2" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40365 } } peers { permanent_uuid: "9728b0a476874f8d84efd61d0b15a7eb" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 40377 } attrs { promote: false } } peers { permanent_uuid: "61915ec9e9cd43f8b944b77b8ba364f1" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 44455 } attrs { promote: false } } }
I20260928 06:33:46.337666 23691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a9edcc4ed794902a9d229837466efd2 [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:46.337936 23738 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 61915ec9e9cd43f8b944b77b8ba364f1 (127.16.157.124) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "3a9edcc4ed794902a9d229837466efd2" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40365 } } peers { permanent_uuid: "9728b0a476874f8d84efd61d0b15a7eb" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 40377 } attrs { promote: false } } peers { permanent_uuid: "61915ec9e9cd43f8b944b77b8ba364f1" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 44455 } attrs { promote: false } } }
I20260928 06:33:46.339048 23926 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 61915ec9e9cd43f8b944b77b8ba364f1 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 61915ec9e9cd43f8b944b77b8ba364f1 (127.16.157.124) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "3a9edcc4ed794902a9d229837466efd2" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40365 } } peers { permanent_uuid: "9728b0a476874f8d84efd61d0b15a7eb" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 40377 } attrs { promote: false } } peers { permanent_uuid: "61915ec9e9cd43f8b944b77b8ba364f1" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 44455 } attrs { promote: false } } }
I20260928 06:33:46.340173 24419 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3a9edcc4ed794902a9d229837466efd2" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "3a9edcc4ed794902a9d229837466efd2" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40365 } } peers { permanent_uuid: "9728b0a476874f8d84efd61d0b15a7eb" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 40377 } attrs { promote: false } } peers { permanent_uuid: "61915ec9e9cd43f8b944b77b8ba364f1" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 44455 } attrs { promote: false } } }
I20260928 06:33:46.340272 24419 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:33:46.340307 23955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61915ec9e9cd43f8b944b77b8ba364f1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3a9edcc4ed794902a9d229837466efd2" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "3a9edcc4ed794902a9d229837466efd2" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40365 } } peers { permanent_uuid: "9728b0a476874f8d84efd61d0b15a7eb" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 40377 } attrs { promote: false } } peers { permanent_uuid: "61915ec9e9cd43f8b944b77b8ba364f1" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 44455 } attrs { promote: false } } }
I20260928 06:33:46.340374 23955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61915ec9e9cd43f8b944b77b8ba364f1 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:33:46.341173 24397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:46.338103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54220 (local address 127.16.157.65:36871)
0928 06:33:46.338143 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:46.338145 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:46.338254 (+ 109us) server_negotiation.cc:407] Connection header received
0928 06:33:46.338288 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:46.338289 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:46.338306 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:46.338369 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:33:46.338556 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:46.339240 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:46.340284 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:46.340589 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:46.340712 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:46.340872 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:33:46.340985 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:33:46.341064 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:46.341087 (+ 23us) server_negotiation.cc:299] Negotiation successful
0928 06:33:46.341110 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:33:46.343088 24189 raft_consensus.cc:1261] T 22bbc5baf6854d27b5320c5a35829061 P ae0e85a2c6794239a27fd60294c0a03f [term 1 FOLLOWER]: Refusing update from remote peer d8d60b2a738a49288570af1432d7c119: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:33:46.343098 24326 raft_consensus.cc:1261] T 22bbc5baf6854d27b5320c5a35829061 P 6cec1f1c72de4f28a819e930d1f014b5 [term 1 FOLLOWER]: Refusing update from remote peer d8d60b2a738a49288570af1432d7c119: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:33:46.343493 24415 consensus_queue.cc:1035] T 22bbc5baf6854d27b5320c5a35829061 P d8d60b2a738a49288570af1432d7c119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0e85a2c6794239a27fd60294c0a03f" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 39405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:33:46.343611 24408 consensus_queue.cc:1035] T 22bbc5baf6854d27b5320c5a35829061 P d8d60b2a738a49288570af1432d7c119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cec1f1c72de4f28a819e930d1f014b5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 33283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:33:46.349758 23117 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:33:46.349849 23117 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:33:46.350142 24420 mvcc.cc:205] Tried to move back new op lower bound from 7334204319098572800 to 7334204319058993152. Current Snapshot: MvccSnapshot[applied={T|T < 7334204319098572800 or (T in {7334204319098572800})}]
I20260928 06:33:46.351135 23117 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3a9edcc4ed794902a9d229837466efd2: Sending DeleteTablet for 3 replicas of tablet 22bbc5baf6854d27b5320c5a35829061
I20260928 06:33:46.351447 24306 tablet_service.cc:1514] Processing DeleteTablet for tablet 22bbc5baf6854d27b5320c5a35829061 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:33:46 UTC) from {username='oryx', principal='oryx/127.16.157.126@KRBTEST.COM'} at 127.0.0.1:35872
I20260928 06:33:46.351526 24029 tablet_service.cc:1514] Processing DeleteTablet for tablet 22bbc5baf6854d27b5320c5a35829061 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:33:46 UTC) from {username='oryx', principal='oryx/127.16.157.126@KRBTEST.COM'} at 127.0.0.1:54214
I20260928 06:33:46.351663 24427 tablet_replica.cc:330] T 22bbc5baf6854d27b5320c5a35829061 P 6cec1f1c72de4f28a819e930d1f014b5: stopping tablet replica
I20260928 06:33:46.351763 24427 raft_consensus.cc:2229] T 22bbc5baf6854d27b5320c5a35829061 P 6cec1f1c72de4f28a819e930d1f014b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:33:46.351835 24427 raft_consensus.cc:2258] T 22bbc5baf6854d27b5320c5a35829061 P 6cec1f1c72de4f28a819e930d1f014b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:33:46.351888 24168 tablet_service.cc:1514] Processing DeleteTablet for tablet 22bbc5baf6854d27b5320c5a35829061 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:33:46 UTC) from {username='oryx', principal='oryx/127.16.157.126@KRBTEST.COM'} at 127.0.0.1:54086
I20260928 06:33:46.352190 24429 tablet_replica.cc:330] T 22bbc5baf6854d27b5320c5a35829061 P ae0e85a2c6794239a27fd60294c0a03f: stopping tablet replica
I20260928 06:33:46.352322 24429 raft_consensus.cc:2229] T 22bbc5baf6854d27b5320c5a35829061 P ae0e85a2c6794239a27fd60294c0a03f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:33:46.352420 24429 raft_consensus.cc:2258] T 22bbc5baf6854d27b5320c5a35829061 P ae0e85a2c6794239a27fd60294c0a03f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:33:46.352895 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 23086
I20260928 06:33:46.357676 24428 tablet_replica.cc:330] T 22bbc5baf6854d27b5320c5a35829061 P d8d60b2a738a49288570af1432d7c119: stopping tablet replica
I20260928 06:33:46.357788 24428 raft_consensus.cc:2229] T 22bbc5baf6854d27b5320c5a35829061 P d8d60b2a738a49288570af1432d7c119 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:33:46.357885 24428 raft_consensus.cc:2258] T 22bbc5baf6854d27b5320c5a35829061 P d8d60b2a738a49288570af1432d7c119 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:33:46.358386 24428 ts_tablet_manager.cc:1905] T 22bbc5baf6854d27b5320c5a35829061 P d8d60b2a738a49288570af1432d7c119: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:33:46.362241 24428 ts_tablet_manager.cc:1918] T 22bbc5baf6854d27b5320c5a35829061 P d8d60b2a738a49288570af1432d7c119: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:33:46.362706 24429 ts_tablet_manager.cc:1905] T 22bbc5baf6854d27b5320c5a35829061 P ae0e85a2c6794239a27fd60294c0a03f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:33:46.364194 24103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:46.352302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58956 (local address 127.16.157.124:44455)
0928 06:33:46.352345 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:46.352347 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:46.353034 (+ 687us) server_negotiation.cc:407] Connection header received
0928 06:33:46.354381 (+ 1347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:46.354382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:46.354393 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:46.354422 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:46.355613 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:46.356411 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:46.358122 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:46.358474 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:46.358525 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:46.360274 (+ 1749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:46.360275 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:46.360275 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:46.360283 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:46.362284 (+ 2001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:46.363159 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:46.363160 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:46.363160 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:46.363219 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:46.363576 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:46.363577 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:46.363577 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:46.363719 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:46.363801 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:46.364036 (+ 235us) server_negotiation.cc:299] Negotiation successful
0928 06:33:46.364110 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:33:46.365294 24428 log.cc:1196] T 22bbc5baf6854d27b5320c5a35829061 P d8d60b2a738a49288570af1432d7c119: Deleting WAL directory at /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-0/wal/wals/22bbc5baf6854d27b5320c5a35829061
I20260928 06:33:46.365928 24429 ts_tablet_manager.cc:1918] T 22bbc5baf6854d27b5320c5a35829061 P ae0e85a2c6794239a27fd60294c0a03f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:33:46.366012 24429 log.cc:1196] T 22bbc5baf6854d27b5320c5a35829061 P ae0e85a2c6794239a27fd60294c0a03f: Deleting WAL directory at /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-1/wal/wals/22bbc5baf6854d27b5320c5a35829061
I20260928 06:33:46.366344 24429 ts_tablet_manager.cc:1939] T 22bbc5baf6854d27b5320c5a35829061 P ae0e85a2c6794239a27fd60294c0a03f: Deleting consensus metadata
I20260928 06:33:46.366516 24427 ts_tablet_manager.cc:1905] T 22bbc5baf6854d27b5320c5a35829061 P 6cec1f1c72de4f28a819e930d1f014b5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 06:33:46.366716 24122 connection.cc:441] server connection from 127.0.0.1:54086 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54086 (request call id 1) could send its response
I20260928 06:33:46.366885 24428 ts_tablet_manager.cc:1939] T 22bbc5baf6854d27b5320c5a35829061 P d8d60b2a738a49288570af1432d7c119: Deleting consensus metadata
W20260928 06:33:46.367231 23985 connection.cc:441] server connection from 127.0.0.1:54214 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54214 (request call id 1) could send its response
I20260928 06:33:46.368090 24427 ts_tablet_manager.cc:1918] T 22bbc5baf6854d27b5320c5a35829061 P 6cec1f1c72de4f28a819e930d1f014b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:33:46.368168 24427 log.cc:1196] T 22bbc5baf6854d27b5320c5a35829061 P 6cec1f1c72de4f28a819e930d1f014b5: Deleting WAL directory at /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-2/wal/wals/22bbc5baf6854d27b5320c5a35829061
I20260928 06:33:46.368444 24427 ts_tablet_manager.cc:1939] T 22bbc5baf6854d27b5320c5a35829061 P 6cec1f1c72de4f28a819e930d1f014b5: Deleting consensus metadata
W20260928 06:33:46.368777 24259 connection.cc:441] server connection from 127.0.0.1:35872 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35872 (request call id 1) could send its response
I20260928 06:33:46.375900 24103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:46.365224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58960 (local address 127.16.157.124:44455)
0928 06:33:46.365280 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:46.365283 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:46.367523 (+ 2240us) server_negotiation.cc:407] Connection header received
0928 06:33:46.367591 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:46.367592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:46.367603 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:46.367653 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:46.368741 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:46.369507 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:46.370769 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:46.371048 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:46.371090 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:46.372921 (+ 1831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:46.372922 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:46.372922 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:46.372930 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:46.374573 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:46.374989 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:46.374990 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:46.374991 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:46.375048 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:46.375260 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:46.375261 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:46.375261 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:46.375392 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:46.375483 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:46.375706 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 06:33:46.375783 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 06:33:46.377017 24384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:46.364900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33920 (local address 127.16.157.125:40377)
0928 06:33:46.364940 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:46.364942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:46.367402 (+ 2460us) server_negotiation.cc:407] Connection header received
0928 06:33:46.367678 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:46.367679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:46.367692 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:46.367724 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:46.368991 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:46.369732 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:46.370751 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:46.371065 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:46.371101 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:46.372921 (+ 1820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:46.372922 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:46.372922 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:46.372930 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:46.375517 (+ 2587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:46.375879 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:46.375880 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:46.375880 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:46.375934 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:46.376169 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:46.376169 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:46.376170 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:46.376292 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:46.376392 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:46.376809 (+ 417us) server_negotiation.cc:299] Negotiation successful
0928 06:33:46.376899 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 06:33:47.347836 24440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:47.347552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40365 (local address 127.16.157.66:47529)
0928 06:33:47.347680 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:47.347743 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:40365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":70,"threads_started":1}
W20260928 06:33:47.348275 24236 heartbeater.cc:644] Failed to heartbeat to 127.16.157.126:40365 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.157.126:40365: connect: Connection refused (error 111)
I20260928 06:33:47.348626 24441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:47.348411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40365 (local address 127.16.157.67:55519)
0928 06:33:47.348512 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:47.348591 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:40365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":53,"threads_started":1}
W20260928 06:33:47.348846 24373 heartbeater.cc:644] Failed to heartbeat to 127.16.157.126:40365 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.157.126:40365: connect: Connection refused (error 111)
I20260928 06:33:47.350243 24442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:47.349922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40365 (local address 127.16.157.65:47665)
0928 06:33:47.350137 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:47.350200 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:40365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":66,"threads_started":1}
W20260928 06:33:47.350422 24097 heartbeater.cc:644] Failed to heartbeat to 127.16.157.126:40365 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.157.126:40365: connect: Connection refused (error 111)
I20260928 06:33:47.851599 24443 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a9edcc4ed794902a9d229837466efd2)
I20260928 06:33:47.851763 24443 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "3a9edcc4ed794902a9d229837466efd2" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40365 } } peers { permanent_uuid: "9728b0a476874f8d84efd61d0b15a7eb" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 40377 } attrs { promote: false } } peers { permanent_uuid: "61915ec9e9cd43f8b944b77b8ba364f1" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 44455 } attrs { promote: false } }
I20260928 06:33:47.852074 24443 leader_election.cc:288] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a9edcc4ed794902a9d229837466efd2 (127.16.157.126:40365), 61915ec9e9cd43f8b944b77b8ba364f1 (127.16.157.124:44455)
I20260928 06:33:47.852492 24444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:47.852205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40365 (local address 127.0.0.1:49302)
0928 06:33:47.852364 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:47.852432 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:40365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":54,"threads_started":1}
W20260928 06:33:47.852672 23713 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.157.126:40365: connect: Connection refused (error 111)
I20260928 06:33:47.853027 24448 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61915ec9e9cd43f8b944b77b8ba364f1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a9edcc4ed794902a9d229837466efd2)
I20260928 06:33:47.853144 24448 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61915ec9e9cd43f8b944b77b8ba364f1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "3a9edcc4ed794902a9d229837466efd2" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40365 } } peers { permanent_uuid: "9728b0a476874f8d84efd61d0b15a7eb" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 40377 } attrs { promote: false } } peers { permanent_uuid: "61915ec9e9cd43f8b944b77b8ba364f1" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 44455 } attrs { promote: false } }
I20260928 06:33:47.853201 24444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:47.853056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40365 (local address 127.0.0.1:49312)
0928 06:33:47.853142 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:47.853174 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:40365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260928 06:33:47.853323 23713 leader_election.cc:334] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3a9edcc4ed794902a9d229837466efd2 (127.16.157.126:40365): Network error: Client connection negotiation failed: client connection to 127.16.157.126:40365: connect: Connection refused (error 111)
I20260928 06:33:47.853633 24448 leader_election.cc:288] T 00000000000000000000000000000000 P 61915ec9e9cd43f8b944b77b8ba364f1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a9edcc4ed794902a9d229837466efd2 (127.16.157.126:40365), 9728b0a476874f8d84efd61d0b15a7eb (127.16.157.125:40377)
I20260928 06:33:47.853747 24449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:47.853472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40365 (local address 127.0.0.1:49324)
0928 06:33:47.853602 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:47.853659 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:40365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":82,"threads_started":1}
W20260928 06:33:47.853878 23901 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.157.126:40365: connect: Connection refused (error 111)
I20260928 06:33:47.854400 24449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:47.854290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40365 (local address 127.0.0.1:49334)
0928 06:33:47.854343 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:47.854375 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:40365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 06:33:47.854480 23901 leader_election.cc:334] T 00000000000000000000000000000000 P 61915ec9e9cd43f8b944b77b8ba364f1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3a9edcc4ed794902a9d229837466efd2 (127.16.157.126:40365): Network error: Client connection negotiation failed: client connection to 127.16.157.126:40365: connect: Connection refused (error 111)
I20260928 06:33:47.856077 24445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:47.852378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.124:44455 (local address 127.0.0.1:58964)
0928 06:33:47.852483 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:47.852494 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:47.852551 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:47.852817 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:47.852818 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:47.852821 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:33:47.852894 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:47.852897 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:47.854197 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:47.854199 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:47.855176 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:47.855202 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:47.855885 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:47.855886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:47.855934 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:47.855938 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:33:47.855994 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 06:33:47.856030 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":51,"threads_started":1}
I20260928 06:33:47.856117 24446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:47.852516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58964 (local address 127.16.157.124:44455)
0928 06:33:47.852633 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:47.852635 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:47.852647 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:33:47.852677 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:47.852678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:47.852691 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:47.852800 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:33:47.853019 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:47.854112 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:47.855288 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:47.855800 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:47.855886 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:47.855895 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:47.856014 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 06:33:47.856057 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":58,"threads_started":1}
I20260928 06:33:47.856416 23926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9728b0a476874f8d84efd61d0b15a7eb" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "61915ec9e9cd43f8b944b77b8ba364f1" is_pre_election: true
I20260928 06:33:47.856520 23926 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 61915ec9e9cd43f8b944b77b8ba364f1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9728b0a476874f8d84efd61d0b15a7eb in term 1.
I20260928 06:33:47.856681 23713 leader_election.cc:302] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61915ec9e9cd43f8b944b77b8ba364f1, 9728b0a476874f8d84efd61d0b15a7eb; no voters: 3a9edcc4ed794902a9d229837466efd2
I20260928 06:33:47.856787 24443 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 06:33:47.856837 24443 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3a9edcc4ed794902a9d229837466efd2)
I20260928 06:33:47.856871 24443 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb [term 1 FOLLOWER]: Advancing to term 2
I20260928 06:33:47.857342 24450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:47.853747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.125:40377 (local address 127.0.0.1:33922)
0928 06:33:47.853983 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:47.854013 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:47.854098 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:47.854342 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:47.854343 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:47.854345 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:33:47.854393 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:47.854394 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:47.855109 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:47.855110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:47.856297 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:47.856300 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:47.857101 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:47.857102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:47.857194 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:47.857198 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:33:47.857255 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 06:33:47.857298 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":56,"threads_started":1}
I20260928 06:33:47.857373 24451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:47.853803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33922 (local address 127.16.157.125:40377)
0928 06:33:47.853920 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:47.853922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:47.854104 (+ 182us) server_negotiation.cc:407] Connection header received
0928 06:33:47.854158 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:47.854159 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:47.854170 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:47.854242 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:33:47.854501 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:47.855025 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:47.856386 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:47.856984 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:47.857099 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:47.857107 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:47.857270 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 06:33:47.857309 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":71,"threads_started":1}
I20260928 06:33:47.857647 23738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "61915ec9e9cd43f8b944b77b8ba364f1" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9728b0a476874f8d84efd61d0b15a7eb" is_pre_election: true
I20260928 06:33:47.858230 24443 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "3a9edcc4ed794902a9d229837466efd2" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40365 } } peers { permanent_uuid: "9728b0a476874f8d84efd61d0b15a7eb" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 40377 } attrs { promote: false } } peers { permanent_uuid: "61915ec9e9cd43f8b944b77b8ba364f1" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 44455 } attrs { promote: false } }
I20260928 06:33:47.858354 23738 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 61915ec9e9cd43f8b944b77b8ba364f1 in current term 2: Already voted for candidate 9728b0a476874f8d84efd61d0b15a7eb in this term.
I20260928 06:33:47.858371 24443 leader_election.cc:288] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb [CANDIDATE]: Term 2 election: Requested vote from peers 3a9edcc4ed794902a9d229837466efd2 (127.16.157.126:40365), 61915ec9e9cd43f8b944b77b8ba364f1 (127.16.157.124:44455)
I20260928 06:33:47.858650 24445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:47.858503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40365 (local address 127.0.0.1:49356)
0928 06:33:47.858586 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:47.858611 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:40365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260928 06:33:47.858641 23903 leader_election.cc:302] T 00000000000000000000000000000000 P 61915ec9e9cd43f8b944b77b8ba364f1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61915ec9e9cd43f8b944b77b8ba364f1; no voters: 3a9edcc4ed794902a9d229837466efd2, 9728b0a476874f8d84efd61d0b15a7eb
I20260928 06:33:47.858769 24448 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61915ec9e9cd43f8b944b77b8ba364f1 [term 1 FOLLOWER]: Advancing to term 2
I20260928 06:33:47.858865 23926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9728b0a476874f8d84efd61d0b15a7eb" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "61915ec9e9cd43f8b944b77b8ba364f1"
I20260928 06:33:47.858982 24445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:47.858900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40365 (local address 127.0.0.1:49358)
0928 06:33:47.858935 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:47.858959 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:40365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260928 06:33:47.859082 23713 leader_election.cc:334] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3a9edcc4ed794902a9d229837466efd2 (127.16.157.126:40365): Network error: Client connection negotiation failed: client connection to 127.16.157.126:40365: connect: Connection refused (error 111)
I20260928 06:33:47.859920 24448 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 61915ec9e9cd43f8b944b77b8ba364f1 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:33:47.860651 23926 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 61915ec9e9cd43f8b944b77b8ba364f1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9728b0a476874f8d84efd61d0b15a7eb in term 2.
I20260928 06:33:47.861136 23713 leader_election.cc:302] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61915ec9e9cd43f8b944b77b8ba364f1, 9728b0a476874f8d84efd61d0b15a7eb; no voters: 3a9edcc4ed794902a9d229837466efd2
I20260928 06:33:47.861235 24443 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb [term 2 FOLLOWER]: Leader election won for term 2
I20260928 06:33:47.861405 24443 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb [term 2 LEADER]: Becoming Leader. State: Replica: 9728b0a476874f8d84efd61d0b15a7eb, State: Running, Role: LEADER
I20260928 06:33:47.861481 24443 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "3a9edcc4ed794902a9d229837466efd2" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40365 } } peers { permanent_uuid: "9728b0a476874f8d84efd61d0b15a7eb" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 40377 } attrs { promote: false } } peers { permanent_uuid: "61915ec9e9cd43f8b944b77b8ba364f1" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 44455 } attrs { promote: false } }
I20260928 06:33:47.861800 24453 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9728b0a476874f8d84efd61d0b15a7eb. Latest consensus state: current_term: 2 leader_uuid: "9728b0a476874f8d84efd61d0b15a7eb" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "3a9edcc4ed794902a9d229837466efd2" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40365 } } peers { permanent_uuid: "9728b0a476874f8d84efd61d0b15a7eb" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 40377 } attrs { promote: false } } peers { permanent_uuid: "61915ec9e9cd43f8b944b77b8ba364f1" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 44455 } attrs { promote: false } } }
I20260928 06:33:47.861932 24453 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:47.862116 24455 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:33:47.862632 24455 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:33:47.862731 24455 catalog_manager.cc:1259] Loaded cluster ID: fc73c8eec1414312b3fe0164efd101a8
I20260928 06:33:47.862776 24455 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:33:47.862957 24455 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:33:47.863061 24455 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb: Loaded TSK: 0
I20260928 06:33:47.863229 24455 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:33:47.915491 23727 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33920:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:33:47.917706 23727 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:33:47.918889 24445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:47.918748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40365 (local address 127.0.0.1:49376)
0928 06:33:47.918805 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:47.918854 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:40365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 06:33:47.918985 23926 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 61915ec9e9cd43f8b944b77b8ba364f1 [term 2 FOLLOWER]: Refusing update from remote peer 9728b0a476874f8d84efd61d0b15a7eb: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260928 06:33:47.919245 24443 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "61915ec9e9cd43f8b944b77b8ba364f1" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 44455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260928 06:33:47.919397 24445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:47.919295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40365 (local address 127.0.0.1:49378)
0928 06:33:47.919346 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:47.919376 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:40365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 06:33:47.919518 23713 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb -> Peer 3a9edcc4ed794902a9d229837466efd2 (127.16.157.126:40365): Couldn't send request to peer 3a9edcc4ed794902a9d229837466efd2. Status: Network error: Client connection negotiation failed: client connection to 127.16.157.126:40365: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 06:33:47.920013 24448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61915ec9e9cd43f8b944b77b8ba364f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9728b0a476874f8d84efd61d0b15a7eb. Latest consensus state: current_term: 2 leader_uuid: "9728b0a476874f8d84efd61d0b15a7eb" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "3a9edcc4ed794902a9d229837466efd2" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40365 } } peers { permanent_uuid: "9728b0a476874f8d84efd61d0b15a7eb" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 40377 } attrs { promote: false } } peers { permanent_uuid: "61915ec9e9cd43f8b944b77b8ba364f1" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 44455 } attrs { promote: false } } }
I20260928 06:33:47.920092 24448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61915ec9e9cd43f8b944b77b8ba364f1 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:33:47.920614 24443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "9728b0a476874f8d84efd61d0b15a7eb" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "3a9edcc4ed794902a9d229837466efd2" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40365 } } peers { permanent_uuid: "9728b0a476874f8d84efd61d0b15a7eb" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 40377 } attrs { promote: false } } peers { permanent_uuid: "61915ec9e9cd43f8b944b77b8ba364f1" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 44455 } attrs { promote: false } } }
I20260928 06:33:47.920707 24443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:47.920779 24457 mvcc.cc:205] Tried to move back new op lower bound from 7334204325553508352 to 7334204325321072640. Current Snapshot: MvccSnapshot[applied={T|T < 7334204325553508352}]
I20260928 06:33:47.920924 24448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61915ec9e9cd43f8b944b77b8ba364f1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "9728b0a476874f8d84efd61d0b15a7eb" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "3a9edcc4ed794902a9d229837466efd2" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40365 } } peers { permanent_uuid: "9728b0a476874f8d84efd61d0b15a7eb" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 40377 } attrs { promote: false } } peers { permanent_uuid: "61915ec9e9cd43f8b944b77b8ba364f1" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 44455 } attrs { promote: false } } }
I20260928 06:33:47.921011 24448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61915ec9e9cd43f8b944b77b8ba364f1 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:33:47.925823 24463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:47.922443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.66:39405 (local address 127.0.0.1:54098)
0928 06:33:47.922680 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:47.922691 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:47.922731 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:47.922984 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:47.922985 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:47.922987 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:33:47.923058 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:47.923060 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:47.923834 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:47.923849 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:47.924756 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:47.924834 (+ 78us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:47.925615 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:47.925616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:47.925672 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:47.925676 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:33:47.925720 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 06:33:47.925760 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":20}
I20260928 06:33:47.926139 24444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:47.922553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.65:36871 (local address 127.0.0.1:54232)
0928 06:33:47.922802 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:47.922811 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:47.922924 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:47.923156 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:47.923157 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:47.923158 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:33:47.923200 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:47.923201 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:47.924065 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:47.924066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:47.925235 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:47.925238 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:47.925955 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:47.925956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:47.926013 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:47.926016 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:33:47.926044 (+ 28us) client_negotiation.cc:240] Negotiation successful
0928 06:33:47.926083 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":69,"threads_started":1}
I20260928 06:33:47.926404 24464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:47.922607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54232 (local address 127.16.157.65:36871)
0928 06:33:47.922734 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:47.922736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:47.922856 (+ 120us) server_negotiation.cc:407] Connection header received
0928 06:33:47.923065 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:47.923066 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:47.923077 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:47.923142 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:33:47.923252 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:47.923989 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:47.925306 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:47.925858 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:47.926255 (+ 397us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:47.926269 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:47.926287 (+ 18us) server_negotiation.cc:299] Negotiation successful
0928 06:33:47.926329 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":74,"threads_started":1}
I20260928 06:33:47.926596 24029 tablet_service.cc:1467] Processing CreateTablet for tablet d4a4458bb80c4d0b99d396bc51827223 (DEFAULT_TABLE table=test-table [id=e13ababaab374813a015f2ca0c38c9bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:33:47.926738 24029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4a4458bb80c4d0b99d396bc51827223. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:47.927014 24465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:47.922700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54098 (local address 127.16.157.66:39405)
0928 06:33:47.922833 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:47.922835 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:47.922843 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:33:47.922871 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:47.922872 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:47.922881 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:47.922939 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:33:47.923136 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:47.923742 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:47.924931 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:47.925490 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:47.925554 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:47.925565 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:47.926087 (+ 522us) server_negotiation.cc:299] Negotiation successful
0928 06:33:47.926130 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":66,"threads_started":1}
I20260928 06:33:47.927227 24168 tablet_service.cc:1467] Processing CreateTablet for tablet d4a4458bb80c4d0b99d396bc51827223 (DEFAULT_TABLE table=test-table [id=e13ababaab374813a015f2ca0c38c9bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:33:47.927350 24168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4a4458bb80c4d0b99d396bc51827223. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:47.928218 24466 tablet_bootstrap.cc:492] T d4a4458bb80c4d0b99d396bc51827223 P d8d60b2a738a49288570af1432d7c119: Bootstrap starting.
I20260928 06:33:47.928642 24466 tablet_bootstrap.cc:654] T d4a4458bb80c4d0b99d396bc51827223 P d8d60b2a738a49288570af1432d7c119: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:47.928809 24467 tablet_bootstrap.cc:492] T d4a4458bb80c4d0b99d396bc51827223 P ae0e85a2c6794239a27fd60294c0a03f: Bootstrap starting.
I20260928 06:33:47.929167 24467 tablet_bootstrap.cc:654] T d4a4458bb80c4d0b99d396bc51827223 P ae0e85a2c6794239a27fd60294c0a03f: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:47.929244 24466 tablet_bootstrap.cc:492] T d4a4458bb80c4d0b99d396bc51827223 P d8d60b2a738a49288570af1432d7c119: No bootstrap required, opened a new log
I20260928 06:33:47.929297 24466 ts_tablet_manager.cc:1397] T d4a4458bb80c4d0b99d396bc51827223 P d8d60b2a738a49288570af1432d7c119: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:47.929430 24466 raft_consensus.cc:348] T d4a4458bb80c4d0b99d396bc51827223 P d8d60b2a738a49288570af1432d7c119 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0e85a2c6794239a27fd60294c0a03f" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 39405 } } peers { permanent_uuid: "d8d60b2a738a49288570af1432d7c119" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 36871 } } peers { permanent_uuid: "6cec1f1c72de4f28a819e930d1f014b5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 33283 } }
I20260928 06:33:47.929509 24466 raft_consensus.cc:374] T d4a4458bb80c4d0b99d396bc51827223 P d8d60b2a738a49288570af1432d7c119 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:47.929538 24466 raft_consensus.cc:729] T d4a4458bb80c4d0b99d396bc51827223 P d8d60b2a738a49288570af1432d7c119 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8d60b2a738a49288570af1432d7c119, State: Initialized, Role: FOLLOWER
I20260928 06:33:47.929600 24466 consensus_queue.cc:260] T d4a4458bb80c4d0b99d396bc51827223 P d8d60b2a738a49288570af1432d7c119 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0e85a2c6794239a27fd60294c0a03f" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 39405 } } peers { permanent_uuid: "d8d60b2a738a49288570af1432d7c119" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 36871 } } peers { permanent_uuid: "6cec1f1c72de4f28a819e930d1f014b5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 33283 } }
I20260928 06:33:47.929704 24445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:47.922442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.67:33283 (local address 127.0.0.1:35886)
0928 06:33:47.922489 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:47.922498 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:47.922529 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:47.923432 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:47.923432 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:47.923434 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:33:47.923465 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:47.923466 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:47.925691 (+ 2225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:47.925692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:47.926869 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:47.926872 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:47.929414 (+ 2542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:47.929415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:47.929478 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:47.929483 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:33:47.929554 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 06:33:47.929613 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 06:33:47.929778 24466 ts_tablet_manager.cc:1428] T d4a4458bb80c4d0b99d396bc51827223 P d8d60b2a738a49288570af1432d7c119: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 06:33:47.930234 24471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:47.930041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40365 (local address 127.16.157.65:43669)
0928 06:33:47.930166 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:47.930204 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:40365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":79,"threads_started":1}
I20260928 06:33:47.930369 24462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:47.922507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35886 (local address 127.16.157.67:33283)
0928 06:33:47.923306 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:47.923309 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:47.923316 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:33:47.923340 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:47.923341 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:47.923351 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:47.923412 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:33:47.923544 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:47.925605 (+ 2061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:47.926977 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:47.929307 (+ 2330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:47.929994 (+ 687us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:47.930010 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:47.930030 (+ 20us) server_negotiation.cc:299] Negotiation successful
0928 06:33:47.930290 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":777,"thread_start_us":64,"threads_started":1}
I20260928 06:33:47.930572 24306 tablet_service.cc:1467] Processing CreateTablet for tablet d4a4458bb80c4d0b99d396bc51827223 (DEFAULT_TABLE table=test-table [id=e13ababaab374813a015f2ca0c38c9bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:33:47.930699 24306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4a4458bb80c4d0b99d396bc51827223. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:47.930991 24471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:47.930822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40365 (local address 127.16.157.65:57533)
0928 06:33:47.930931 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:47.930967 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:40365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260928 06:33:47.930994 24467 tablet_bootstrap.cc:492] T d4a4458bb80c4d0b99d396bc51827223 P ae0e85a2c6794239a27fd60294c0a03f: No bootstrap required, opened a new log
I20260928 06:33:47.931042 24467 ts_tablet_manager.cc:1397] T d4a4458bb80c4d0b99d396bc51827223 P ae0e85a2c6794239a27fd60294c0a03f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:33:47.931164 24467 raft_consensus.cc:348] T d4a4458bb80c4d0b99d396bc51827223 P ae0e85a2c6794239a27fd60294c0a03f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0e85a2c6794239a27fd60294c0a03f" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 39405 } } peers { permanent_uuid: "d8d60b2a738a49288570af1432d7c119" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 36871 } } peers { permanent_uuid: "6cec1f1c72de4f28a819e930d1f014b5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 33283 } }
I20260928 06:33:47.931238 24467 raft_consensus.cc:374] T d4a4458bb80c4d0b99d396bc51827223 P ae0e85a2c6794239a27fd60294c0a03f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:47.931257 24467 raft_consensus.cc:729] T d4a4458bb80c4d0b99d396bc51827223 P ae0e85a2c6794239a27fd60294c0a03f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae0e85a2c6794239a27fd60294c0a03f, State: Initialized, Role: FOLLOWER
I20260928 06:33:47.931305 24467 consensus_queue.cc:260] T d4a4458bb80c4d0b99d396bc51827223 P ae0e85a2c6794239a27fd60294c0a03f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0e85a2c6794239a27fd60294c0a03f" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 39405 } } peers { permanent_uuid: "d8d60b2a738a49288570af1432d7c119" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 36871 } } peers { permanent_uuid: "6cec1f1c72de4f28a819e930d1f014b5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 33283 } }
I20260928 06:33:47.931464 24467 ts_tablet_manager.cc:1428] T d4a4458bb80c4d0b99d396bc51827223 P ae0e85a2c6794239a27fd60294c0a03f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 06:33:47.931874 24473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:47.931669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40365 (local address 127.16.157.66:53485)
0928 06:33:47.931793 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:47.931829 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:40365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":81,"threads_started":1}
I20260928 06:33:47.932276 24474 tablet_bootstrap.cc:492] T d4a4458bb80c4d0b99d396bc51827223 P 6cec1f1c72de4f28a819e930d1f014b5: Bootstrap starting.
I20260928 06:33:47.932641 24237 heartbeater.cc:503] Master 127.16.157.125:40377 was elected leader, sending a full tablet report...
I20260928 06:33:47.932704 24474 tablet_bootstrap.cc:654] T d4a4458bb80c4d0b99d396bc51827223 P 6cec1f1c72de4f28a819e930d1f014b5: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:47.932798 24473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:47.932703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40365 (local address 127.16.157.66:57917)
0928 06:33:47.932744 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:47.932768 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:40365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 06:33:47.933409 24474 tablet_bootstrap.cc:492] T d4a4458bb80c4d0b99d396bc51827223 P 6cec1f1c72de4f28a819e930d1f014b5: No bootstrap required, opened a new log
I20260928 06:33:47.933480 24474 ts_tablet_manager.cc:1397] T d4a4458bb80c4d0b99d396bc51827223 P 6cec1f1c72de4f28a819e930d1f014b5: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:47.933640 24474 raft_consensus.cc:348] T d4a4458bb80c4d0b99d396bc51827223 P 6cec1f1c72de4f28a819e930d1f014b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0e85a2c6794239a27fd60294c0a03f" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 39405 } } peers { permanent_uuid: "d8d60b2a738a49288570af1432d7c119" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 36871 } } peers { permanent_uuid: "6cec1f1c72de4f28a819e930d1f014b5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 33283 } }
I20260928 06:33:47.933825 24474 raft_consensus.cc:374] T d4a4458bb80c4d0b99d396bc51827223 P 6cec1f1c72de4f28a819e930d1f014b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:47.933857 24474 raft_consensus.cc:729] T d4a4458bb80c4d0b99d396bc51827223 P 6cec1f1c72de4f28a819e930d1f014b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cec1f1c72de4f28a819e930d1f014b5, State: Initialized, Role: FOLLOWER
I20260928 06:33:47.933918 24474 consensus_queue.cc:260] T d4a4458bb80c4d0b99d396bc51827223 P 6cec1f1c72de4f28a819e930d1f014b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0e85a2c6794239a27fd60294c0a03f" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 39405 } } peers { permanent_uuid: "d8d60b2a738a49288570af1432d7c119" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 36871 } } peers { permanent_uuid: "6cec1f1c72de4f28a819e930d1f014b5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 33283 } }
I20260928 06:33:47.934062 24474 ts_tablet_manager.cc:1428] T d4a4458bb80c4d0b99d396bc51827223 P 6cec1f1c72de4f28a819e930d1f014b5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:47.934659 24477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:47.934273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40365 (local address 127.16.157.67:44667)
0928 06:33:47.934398 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:47.934623 (+ 225us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:40365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":86,"threads_started":1}
I20260928 06:33:47.935120 24477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:47.934977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40365 (local address 127.16.157.67:49009)
0928 06:33:47.935044 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:47.935073 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:40365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 06:33:48.159797 24375 tablet.cc:2369] T d4a4458bb80c4d0b99d396bc51827223 P 6cec1f1c72de4f28a819e930d1f014b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:33:48.193765 24476 raft_consensus.cc:482] T d4a4458bb80c4d0b99d396bc51827223 P 6cec1f1c72de4f28a819e930d1f014b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:33:48.193888 24476 raft_consensus.cc:504] T d4a4458bb80c4d0b99d396bc51827223 P 6cec1f1c72de4f28a819e930d1f014b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0e85a2c6794239a27fd60294c0a03f" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 39405 } } peers { permanent_uuid: "d8d60b2a738a49288570af1432d7c119" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 36871 } } peers { permanent_uuid: "6cec1f1c72de4f28a819e930d1f014b5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 33283 } }
I20260928 06:33:48.194231 24476 leader_election.cc:288] T d4a4458bb80c4d0b99d396bc51827223 P 6cec1f1c72de4f28a819e930d1f014b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0e85a2c6794239a27fd60294c0a03f (127.16.157.66:39405), d8d60b2a738a49288570af1432d7c119 (127.16.157.65:36871)
I20260928 06:33:48.197553 24477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:48.194412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.66:39405 (local address 127.16.157.67:45535)
0928 06:33:48.194651 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:48.194665 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:48.194753 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:48.194979 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:48.194980 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:48.194982 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:33:48.195062 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:48.195064 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:48.195978 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:48.195979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:48.196754 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:48.196756 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:48.197276 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:48.197277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:48.197340 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:48.197369 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 06:33:48.197441 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 06:33:48.197478 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":2}
I20260928 06:33:48.197556 24465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:48.194586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:45535 (local address 127.16.157.66:39405)
0928 06:33:48.194631 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:48.194633 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:48.194754 (+ 121us) server_negotiation.cc:407] Connection header received
0928 06:33:48.194800 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:48.194801 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:48.194812 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:48.194919 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:33:48.195159 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:48.195885 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:48.196830 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:48.197194 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:48.197253 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:48.197260 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:48.197461 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 06:33:48.197501 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 06:33:48.197788 24189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4a4458bb80c4d0b99d396bc51827223" candidate_uuid: "6cec1f1c72de4f28a819e930d1f014b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0e85a2c6794239a27fd60294c0a03f" is_pre_election: true
I20260928 06:33:48.197897 24189 raft_consensus.cc:2454] T d4a4458bb80c4d0b99d396bc51827223 P ae0e85a2c6794239a27fd60294c0a03f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cec1f1c72de4f28a819e930d1f014b5 in term 0.
I20260928 06:33:48.198092 24258 leader_election.cc:302] T d4a4458bb80c4d0b99d396bc51827223 P 6cec1f1c72de4f28a819e930d1f014b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cec1f1c72de4f28a819e930d1f014b5, ae0e85a2c6794239a27fd60294c0a03f; no voters:
I20260928 06:33:48.198215 24476 raft_consensus.cc:2781] T d4a4458bb80c4d0b99d396bc51827223 P 6cec1f1c72de4f28a819e930d1f014b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:33:48.198272 24476 raft_consensus.cc:482] T d4a4458bb80c4d0b99d396bc51827223 P 6cec1f1c72de4f28a819e930d1f014b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:33:48.198347 24476 raft_consensus.cc:3037] T d4a4458bb80c4d0b99d396bc51827223 P 6cec1f1c72de4f28a819e930d1f014b5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:48.198861 24478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:48.194564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.65:36871 (local address 127.16.157.67:35583)
0928 06:33:48.194748 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:48.194759 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:48.194823 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:48.194989 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:48.194991 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:48.194993 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:33:48.195058 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:48.195060 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:48.196840 (+ 1780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:48.196841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:48.197847 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:48.197850 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:48.198624 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:48.198625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:48.198683 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:48.198687 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:33:48.198751 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 06:33:48.198789 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":86,"threads_started":1}
I20260928 06:33:48.198874 24464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:48.194630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:35583 (local address 127.16.157.65:36871)
0928 06:33:48.194683 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:48.194686 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:48.194826 (+ 140us) server_negotiation.cc:407] Connection header received
0928 06:33:48.194908 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:48.194909 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:48.194918 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:48.194967 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:33:48.195124 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:48.195880 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:48.197948 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:48.198526 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:48.198587 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:48.198594 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:48.198761 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 06:33:48.198808 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 06:33:48.199105 24049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4a4458bb80c4d0b99d396bc51827223" candidate_uuid: "6cec1f1c72de4f28a819e930d1f014b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8d60b2a738a49288570af1432d7c119" is_pre_election: true
I20260928 06:33:48.199244 24049 raft_consensus.cc:2454] T d4a4458bb80c4d0b99d396bc51827223 P d8d60b2a738a49288570af1432d7c119 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cec1f1c72de4f28a819e930d1f014b5 in term 0.
I20260928 06:33:48.199535 24476 raft_consensus.cc:504] T d4a4458bb80c4d0b99d396bc51827223 P 6cec1f1c72de4f28a819e930d1f014b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0e85a2c6794239a27fd60294c0a03f" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 39405 } } peers { permanent_uuid: "d8d60b2a738a49288570af1432d7c119" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 36871 } } peers { permanent_uuid: "6cec1f1c72de4f28a819e930d1f014b5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 33283 } }
I20260928 06:33:48.199668 24476 leader_election.cc:288] T d4a4458bb80c4d0b99d396bc51827223 P 6cec1f1c72de4f28a819e930d1f014b5 [CANDIDATE]: Term 1 election: Requested vote from peers ae0e85a2c6794239a27fd60294c0a03f (127.16.157.66:39405), d8d60b2a738a49288570af1432d7c119 (127.16.157.65:36871)
I20260928 06:33:48.199810 24189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4a4458bb80c4d0b99d396bc51827223" candidate_uuid: "6cec1f1c72de4f28a819e930d1f014b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0e85a2c6794239a27fd60294c0a03f"
I20260928 06:33:48.199831 24049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4a4458bb80c4d0b99d396bc51827223" candidate_uuid: "6cec1f1c72de4f28a819e930d1f014b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8d60b2a738a49288570af1432d7c119"
I20260928 06:33:48.199882 24189 raft_consensus.cc:3037] T d4a4458bb80c4d0b99d396bc51827223 P ae0e85a2c6794239a27fd60294c0a03f [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:48.199918 24049 raft_consensus.cc:3037] T d4a4458bb80c4d0b99d396bc51827223 P d8d60b2a738a49288570af1432d7c119 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:48.200642 24189 raft_consensus.cc:2454] T d4a4458bb80c4d0b99d396bc51827223 P ae0e85a2c6794239a27fd60294c0a03f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cec1f1c72de4f28a819e930d1f014b5 in term 1.
I20260928 06:33:48.200692 24049 raft_consensus.cc:2454] T d4a4458bb80c4d0b99d396bc51827223 P d8d60b2a738a49288570af1432d7c119 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cec1f1c72de4f28a819e930d1f014b5 in term 1.
I20260928 06:33:48.200805 24258 leader_election.cc:302] T d4a4458bb80c4d0b99d396bc51827223 P 6cec1f1c72de4f28a819e930d1f014b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cec1f1c72de4f28a819e930d1f014b5, ae0e85a2c6794239a27fd60294c0a03f; no voters:
I20260928 06:33:48.200905 24476 raft_consensus.cc:2781] T d4a4458bb80c4d0b99d396bc51827223 P 6cec1f1c72de4f28a819e930d1f014b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:33:48.201071 24476 raft_consensus.cc:686] T d4a4458bb80c4d0b99d396bc51827223 P 6cec1f1c72de4f28a819e930d1f014b5 [term 1 LEADER]: Becoming Leader. State: Replica: 6cec1f1c72de4f28a819e930d1f014b5, State: Running, Role: LEADER
I20260928 06:33:48.201129 24476 consensus_queue.cc:237] T d4a4458bb80c4d0b99d396bc51827223 P 6cec1f1c72de4f28a819e930d1f014b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0e85a2c6794239a27fd60294c0a03f" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 39405 } } peers { permanent_uuid: "d8d60b2a738a49288570af1432d7c119" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 36871 } } peers { permanent_uuid: "6cec1f1c72de4f28a819e930d1f014b5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 33283 } }
I20260928 06:33:48.201201 24374 heartbeater.cc:503] Master 127.16.157.125:40377 was elected leader, sending a full tablet report...
I20260928 06:33:48.201407 24478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:48.201302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40365 (local address 127.16.157.67:39543)
0928 06:33:48.201340 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:48.201375 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:40365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260928 06:33:48.202001 23728 catalog_manager.cc:5495] T d4a4458bb80c4d0b99d396bc51827223 P 6cec1f1c72de4f28a819e930d1f014b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6cec1f1c72de4f28a819e930d1f014b5 (127.16.157.67). New cstate: current_term: 1 leader_uuid: "6cec1f1c72de4f28a819e930d1f014b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0e85a2c6794239a27fd60294c0a03f" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 39405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8d60b2a738a49288570af1432d7c119" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 36871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6cec1f1c72de4f28a819e930d1f014b5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 33283 } health_report { overall_health: HEALTHY } } }
I20260928 06:33:48.278896 24462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:48.275424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35888 (local address 127.16.157.67:33283)
0928 06:33:48.275480 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:48.275482 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:48.275492 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:33:48.275568 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:48.275569 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:48.275595 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:48.275684 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:33:48.275842 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:48.276652 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:48.278038 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:48.278262 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:48.278313 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:48.278513 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:33:48.278659 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:33:48.278719 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:48.278789 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 06:33:48.278812 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 06:33:48.280174 24189 raft_consensus.cc:1261] T d4a4458bb80c4d0b99d396bc51827223 P ae0e85a2c6794239a27fd60294c0a03f [term 1 FOLLOWER]: Refusing update from remote peer 6cec1f1c72de4f28a819e930d1f014b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:33:48.280269 24049 raft_consensus.cc:1261] T d4a4458bb80c4d0b99d396bc51827223 P d8d60b2a738a49288570af1432d7c119 [term 1 FOLLOWER]: Refusing update from remote peer 6cec1f1c72de4f28a819e930d1f014b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:33:48.280642 24479 consensus_queue.cc:1035] T d4a4458bb80c4d0b99d396bc51827223 P 6cec1f1c72de4f28a819e930d1f014b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d60b2a738a49288570af1432d7c119" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 36871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:33:48.280803 24476 consensus_queue.cc:1035] T d4a4458bb80c4d0b99d396bc51827223 P 6cec1f1c72de4f28a819e930d1f014b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0e85a2c6794239a27fd60294c0a03f" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 39405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:33:48.281589 24098 heartbeater.cc:503] Master 127.16.157.125:40377 was elected leader, sending a full tablet report...
I20260928 06:33:48.281621 24473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:48.281481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40365 (local address 127.16.157.66:33521)
0928 06:33:48.281539 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:48.281588 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:40365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 06:33:48.282009 24471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:48.281864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40365 (local address 127.16.157.65:34375)
0928 06:33:48.281953 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:48.281981 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:40365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260928 06:33:48.282974 24483 mvcc.cc:205] Tried to move back new op lower bound from 7334204327033114624 to 7334204326712414208. Current Snapshot: MvccSnapshot[applied={T|T < 7334204327033114624}]
I20260928 06:33:48.283326 24473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:48.283237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40365 (local address 127.16.157.66:36687)
0928 06:33:48.283273 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:48.283301 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:40365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260928 06:33:48.283620 24471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:48.283526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40365 (local address 127.16.157.65:47791)
0928 06:33:48.283565 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:48.283588 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:40365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260928 06:33:48.284451 24478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:48.283223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40365 (local address 127.16.157.67:39199)
0928 06:33:48.284396 (+ 1173us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:48.284424 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:40365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1156}
I20260928 06:33:48.285305 23728 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:33:48.285398 23728 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:33:48.286792 23728 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9728b0a476874f8d84efd61d0b15a7eb: Sending DeleteTablet for 3 replicas of tablet d4a4458bb80c4d0b99d396bc51827223
I20260928 06:33:48.287134 24306 tablet_service.cc:1514] Processing DeleteTablet for tablet d4a4458bb80c4d0b99d396bc51827223 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:33:48 UTC) from {username='oryx', principal='oryx/127.16.157.125@KRBTEST.COM'} at 127.0.0.1:35886
I20260928 06:33:48.287182 24168 tablet_service.cc:1514] Processing DeleteTablet for tablet d4a4458bb80c4d0b99d396bc51827223 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:33:48 UTC) from {username='oryx', principal='oryx/127.16.157.125@KRBTEST.COM'} at 127.0.0.1:54098
I20260928 06:33:48.287196 24029 tablet_service.cc:1514] Processing DeleteTablet for tablet d4a4458bb80c4d0b99d396bc51827223 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:33:48 UTC) from {username='oryx', principal='oryx/127.16.157.125@KRBTEST.COM'} at 127.0.0.1:54232
I20260928 06:33:48.287467 24492 tablet_replica.cc:330] T d4a4458bb80c4d0b99d396bc51827223 P 6cec1f1c72de4f28a819e930d1f014b5: stopping tablet replica
I20260928 06:33:48.287535 24492 raft_consensus.cc:2229] T d4a4458bb80c4d0b99d396bc51827223 P 6cec1f1c72de4f28a819e930d1f014b5 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:33:48.287619 24492 raft_consensus.cc:2258] T d4a4458bb80c4d0b99d396bc51827223 P 6cec1f1c72de4f28a819e930d1f014b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:33:48.287730 24494 tablet_replica.cc:330] T d4a4458bb80c4d0b99d396bc51827223 P ae0e85a2c6794239a27fd60294c0a03f: stopping tablet replica
I20260928 06:33:48.287781 24494 raft_consensus.cc:2229] T d4a4458bb80c4d0b99d396bc51827223 P ae0e85a2c6794239a27fd60294c0a03f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:33:48.287817 24494 raft_consensus.cc:2258] T d4a4458bb80c4d0b99d396bc51827223 P ae0e85a2c6794239a27fd60294c0a03f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:33:48.287807 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 23875
I20260928 06:33:48.287972 24492 ts_tablet_manager.cc:1905] T d4a4458bb80c4d0b99d396bc51827223 P 6cec1f1c72de4f28a819e930d1f014b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:33:48.289037 24494 ts_tablet_manager.cc:1905] T d4a4458bb80c4d0b99d396bc51827223 P ae0e85a2c6794239a27fd60294c0a03f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:33:48.289649 24492 ts_tablet_manager.cc:1918] T d4a4458bb80c4d0b99d396bc51827223 P 6cec1f1c72de4f28a819e930d1f014b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:33:48.289703 24492 log.cc:1196] T d4a4458bb80c4d0b99d396bc51827223 P 6cec1f1c72de4f28a819e930d1f014b5: Deleting WAL directory at /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-2/wal/wals/d4a4458bb80c4d0b99d396bc51827223
I20260928 06:33:48.289929 24492 ts_tablet_manager.cc:1939] T d4a4458bb80c4d0b99d396bc51827223 P 6cec1f1c72de4f28a819e930d1f014b5: Deleting consensus metadata
I20260928 06:33:48.291792 24494 ts_tablet_manager.cc:1918] T d4a4458bb80c4d0b99d396bc51827223 P ae0e85a2c6794239a27fd60294c0a03f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:33:48.291844 24494 log.cc:1196] T d4a4458bb80c4d0b99d396bc51827223 P ae0e85a2c6794239a27fd60294c0a03f: Deleting WAL directory at /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577196946045-17013-0/minicluster-data/ts-1/wal/wals/d4a4458bb80c4d0b99d396bc51827223
I20260928 06:33:48.292085 24494 ts_tablet_manager.cc:1939] T d4a4458bb80c4d0b99d396bc51827223 P ae0e85a2c6794239a27fd60294c0a03f: Deleting consensus metadata
I20260928 06:33:48.293107 23712 catalog_manager.cc:4841] TS ae0e85a2c6794239a27fd60294c0a03f (127.16.157.66:39405): tablet d4a4458bb80c4d0b99d396bc51827223 (table test-table [id=e13ababaab374813a015f2ca0c38c9bb]) successfully deleted
I20260928 06:33:48.296396 23715 catalog_manager.cc:4841] TS 6cec1f1c72de4f28a819e930d1f014b5 (127.16.157.67:33283): tablet d4a4458bb80c4d0b99d396bc51827223 (table test-table [id=e13ababaab374813a015f2ca0c38c9bb]) successfully deleted
W20260928 06:33:48.296832 24445 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:33:48.296463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.65:36871 (local address 127.0.0.1:54242)
0928 06:33:48.296509 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:48.296519 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:48.296553 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:48.296779 (+ 226us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.65:36871: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 06:33:48.296967 23712 catalog_manager.cc:4584] TS d8d60b2a738a49288570af1432d7c119 (127.16.157.65:36871): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d4a4458bb80c4d0b99d396bc51827223: Network error: Client connection negotiation failed: client connection to 127.16.157.65:36871: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 06:33:48.297731 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 24104
I20260928 06:33:48.304358 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 24242
I20260928 06:33:48.311069 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 23587
I20260928 06:33:48.318444 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 23783
2026-09-28T06:33:48Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5138 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:33:48 dist-test-slave-rrj5 krb5kdc[24497](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:33:48 dist-test-slave-rrj5 krb5kdc[24497](info): set up 2 sockets
Sep 28 06:33:48 dist-test-slave-rrj5 krb5kdc[24497](info): commencing operation
krb5kdc: starting...
W20260928 06:33:49.494930 17013 mini_kdc.cc:121] Time spent starting KDC: real 1.152s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Sep 28 06:33:49 dist-test-slave-rrj5 krb5kdc[24497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:33:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:33:49Z Disabled control of system clock
WARNING: no policy specified for oryx/127.16.157.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.157.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.157.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/krb5kdc/oryx_127.16.157.126.keytab.
Entry for principal oryx/127.16.157.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/krb5kdc/oryx_127.16.157.126.keytab.
I20260928 06:33:49.620059 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:44663
--webserver_interface=127.16.157.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/krb5kdc/oryx_127.16.157.126.keytab
--principal=oryx/127.16.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:37963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.157.126:44663
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:49.712504 24510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:49.712709 24510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:49.712730 24510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:49.713869 24510 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:33:49.713907 24510 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 06:33:49.713927 24510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:49.713999 24510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:49.714018 24510 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:33:49.714039 24510 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:33:49.715653 24510 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:37963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.157.126:44663
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/krb5kdc/oryx_127.16.157.126.keytab
--principal=oryx/127.16.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:44663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:49.715909 24510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:49.716163 24510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:49.720927 24518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:49.720963 24515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:49.720968 24516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:49.721179 24510 server_base.cc:1034] running on GCE node
I20260928 06:33:49.721565 24510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:49.721971 24510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:49.723133 24510 hybrid_clock.cc:648] HybridClock initialized: now 1790577229723116 us; error 35 us; skew 500 ppm
Sep 28 06:33:49 dist-test-slave-rrj5 krb5kdc[24497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577229, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:49.725857 24510 init.cc:377] Logged in from keytab as oryx/127.16.157.126@KRBTEST.COM (short username oryx)
I20260928 06:33:49.726361 24510 webserver.cc:460] Webserver started at http://127.16.157.126:38267/ using document root <none> and password file <none>
I20260928 06:33:49.726593 24510 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:49.726675 24510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:49.726804 24510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:49.727849 24510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/master-0/data/instance:
uuid: "217c529c34ff472987667c1af6f76a6b"
format_stamp: "Formatted at 2026-09-28 06:33:49 on dist-test-slave-rrj5"
server_key: "3ee4dd205bdceb0eb4b61237a7797c70"
server_key_iv: "8aac05e14e8dbb817ebd9a01c95f1be4"
server_key_version: "encryptionkey@0"
I20260928 06:33:49.728253 24510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/master-0/wal/instance:
uuid: "217c529c34ff472987667c1af6f76a6b"
format_stamp: "Formatted at 2026-09-28 06:33:49 on dist-test-slave-rrj5"
server_key: "3ee4dd205bdceb0eb4b61237a7797c70"
server_key_iv: "8aac05e14e8dbb817ebd9a01c95f1be4"
server_key_version: "encryptionkey@0"
I20260928 06:33:49.729640 24510 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:33:49.730443 24525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:49.730672 24510 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:49.730772 24510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/master-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/master-0/wal
uuid: "217c529c34ff472987667c1af6f76a6b"
format_stamp: "Formatted at 2026-09-28 06:33:49 on dist-test-slave-rrj5"
server_key: "3ee4dd205bdceb0eb4b61237a7797c70"
server_key_iv: "8aac05e14e8dbb817ebd9a01c95f1be4"
server_key_version: "encryptionkey@0"
I20260928 06:33:49.730849 24510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:49.740614 24510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:49.743142 24510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:49.743286 24510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:49.748775 24510 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.126:44663
I20260928 06:33:49.748867 24588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.126:44663 every 8 connection(s)
I20260928 06:33:49.749185 24510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
I20260928 06:33:49.749930 24589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:49.752125 24589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 217c529c34ff472987667c1af6f76a6b: Bootstrap starting.
I20260928 06:33:49.752725 24589 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 217c529c34ff472987667c1af6f76a6b: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:49.753039 24589 log.cc:824] T 00000000000000000000000000000000 P 217c529c34ff472987667c1af6f76a6b: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:49.753795 24589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 217c529c34ff472987667c1af6f76a6b: No bootstrap required, opened a new log
I20260928 06:33:49.755445 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 24510
I20260928 06:33:49.755558 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/master-0/wal/instance
I20260928 06:33:49.755753 17013 external_mini_cluster.cc:1428] Setting key 14cef70a71f6c1249e9c381d8d53565a
I20260928 06:33:49.756053 24589 raft_consensus.cc:348] T 00000000000000000000000000000000 P 217c529c34ff472987667c1af6f76a6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "217c529c34ff472987667c1af6f76a6b" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 44663 } }
I20260928 06:33:49.756494 24589 raft_consensus.cc:374] T 00000000000000000000000000000000 P 217c529c34ff472987667c1af6f76a6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:49.756547 24589 raft_consensus.cc:729] T 00000000000000000000000000000000 P 217c529c34ff472987667c1af6f76a6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 217c529c34ff472987667c1af6f76a6b, State: Initialized, Role: FOLLOWER
I20260928 06:33:49.756665 24589 consensus_queue.cc:260] T 00000000000000000000000000000000 P 217c529c34ff472987667c1af6f76a6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "217c529c34ff472987667c1af6f76a6b" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 44663 } }
I20260928 06:33:49.756754 24589 raft_consensus.cc:388] T 00000000000000000000000000000000 P 217c529c34ff472987667c1af6f76a6b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:33:49.756799 24589 raft_consensus.cc:482] T 00000000000000000000000000000000 P 217c529c34ff472987667c1af6f76a6b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:33:49.756852 24589 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 217c529c34ff472987667c1af6f76a6b [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:49.757364 24589 raft_consensus.cc:504] T 00000000000000000000000000000000 P 217c529c34ff472987667c1af6f76a6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "217c529c34ff472987667c1af6f76a6b" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 44663 } }
I20260928 06:33:49.757484 24589 leader_election.cc:302] T 00000000000000000000000000000000 P 217c529c34ff472987667c1af6f76a6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 217c529c34ff472987667c1af6f76a6b; no voters:
I20260928 06:33:49.757695 24589 leader_election.cc:288] T 00000000000000000000000000000000 P 217c529c34ff472987667c1af6f76a6b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:33:49.757750 24594 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 217c529c34ff472987667c1af6f76a6b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:33:49.757910 24594 raft_consensus.cc:686] T 00000000000000000000000000000000 P 217c529c34ff472987667c1af6f76a6b [term 1 LEADER]: Becoming Leader. State: Replica: 217c529c34ff472987667c1af6f76a6b, State: Running, Role: LEADER
I20260928 06:33:49.757969 24589 sys_catalog.cc:564] T 00000000000000000000000000000000 P 217c529c34ff472987667c1af6f76a6b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:33:49.757997 24594 consensus_queue.cc:237] T 00000000000000000000000000000000 P 217c529c34ff472987667c1af6f76a6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "217c529c34ff472987667c1af6f76a6b" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 44663 } }
I20260928 06:33:49.758293 24594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 217c529c34ff472987667c1af6f76a6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "217c529c34ff472987667c1af6f76a6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "217c529c34ff472987667c1af6f76a6b" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 44663 } } }
I20260928 06:33:49.758358 24594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 217c529c34ff472987667c1af6f76a6b [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:49.758471 24594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 217c529c34ff472987667c1af6f76a6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 217c529c34ff472987667c1af6f76a6b. Latest consensus state: current_term: 1 leader_uuid: "217c529c34ff472987667c1af6f76a6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "217c529c34ff472987667c1af6f76a6b" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 44663 } } }
I20260928 06:33:49.758515 24594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 217c529c34ff472987667c1af6f76a6b [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:49.758637 24599 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:33:49.759114 24599 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:33:49.760926 24599 catalog_manager.cc:1347] Generated new cluster ID: d2b215fa5c854c2fbd9241fde99086b6
I20260928 06:33:49.760977 24599 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:33:49 dist-test-slave-rrj5 krb5kdc[24497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM
I20260928 06:33:49.768370 24593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:49.756296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51528 (local address 127.16.157.126:44663)
0928 06:33:49.756479 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:49.756489 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:49.756523 (+ 34us) server_negotiation.cc:407] Connection header received
0928 06:33:49.756649 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:49.756656 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:49.756709 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:49.756852 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:49.757726 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:49.759866 (+ 2140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:49.760931 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:49.761504 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:49.761704 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:49.764400 (+ 2696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:49.764404 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:49.764404 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:49.764416 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:49.766301 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:49.766665 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:49.766667 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:49.766668 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:49.766788 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:49.766957 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:49.766958 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:49.766958 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:49.767221 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:49.767361 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:49.767801 (+ 440us) server_negotiation.cc:299] Negotiation successful
0928 06:33:49.767927 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":103,"threads_started":1}
I20260928 06:33:49.770021 24599 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:33:49.770668 24599 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:33:49.774362 24599 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 217c529c34ff472987667c1af6f76a6b: Generated new TSK 0
I20260928 06:33:49.774538 24599 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.16.157.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.157.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.157.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/krb5kdc/oryx_127.16.157.65.keytab.
Entry for principal oryx/127.16.157.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/krb5kdc/oryx_127.16.157.65.keytab.
I20260928 06:33:49.809962 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.65:0
--local_ip_for_outbound_sockets=127.16.157.65
--webserver_interface=127.16.157.65
--webserver_port=0
--tserver_master_addrs=127.16.157.126:44663
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/krb5kdc/oryx_127.16.157.65.keytab
--principal=oryx/127.16.157.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:37963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:49.921692 24615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:49.921882 24615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:49.921907 24615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:49.923060 24615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:49.923099 24615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:49.923120 24615 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:33:49.923139 24615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.65
I20260928 06:33:49.924633 24615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:37963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/krb5kdc/oryx_127.16.157.65.keytab
--principal=oryx/127.16.157.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.157.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.157.126:44663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.65
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:49.924865 24615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:49.925108 24615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:49.929986 24623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:49.930043 24620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:49.930045 24621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:49.930881 24615 server_base.cc:1034] running on GCE node
I20260928 06:33:49.931070 24615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:49.931298 24615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:49.932461 24615 hybrid_clock.cc:648] HybridClock initialized: now 1790577229932448 us; error 31 us; skew 500 ppm
Sep 28 06:33:49 dist-test-slave-rrj5 krb5kdc[24497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577229, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.157.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:49.938115 24615 init.cc:377] Logged in from keytab as oryx/127.16.157.65@KRBTEST.COM (short username oryx)
I20260928 06:33:49.938647 24615 webserver.cc:460] Webserver started at http://127.16.157.65:36861/ using document root <none> and password file <none>
I20260928 06:33:49.938925 24615 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:49.938992 24615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:49.939131 24615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:49.940562 24615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/ts-0/data/instance:
uuid: "32392e70db0445fe9341ff89e046a25d"
format_stamp: "Formatted at 2026-09-28 06:33:49 on dist-test-slave-rrj5"
server_key: "f0cec47969636d540dc3c60b39f02481"
server_key_iv: "5dc2972c290338c704ad89d7d0b42e8d"
server_key_version: "encryptionkey@0"
I20260928 06:33:49.940976 24615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/ts-0/wal/instance:
uuid: "32392e70db0445fe9341ff89e046a25d"
format_stamp: "Formatted at 2026-09-28 06:33:49 on dist-test-slave-rrj5"
server_key: "f0cec47969636d540dc3c60b39f02481"
server_key_iv: "5dc2972c290338c704ad89d7d0b42e8d"
server_key_version: "encryptionkey@0"
I20260928 06:33:49.942840 24615 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260928 06:33:49.943711 24630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:49.943907 24615 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:49.944007 24615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/ts-0/wal
uuid: "32392e70db0445fe9341ff89e046a25d"
format_stamp: "Formatted at 2026-09-28 06:33:49 on dist-test-slave-rrj5"
server_key: "f0cec47969636d540dc3c60b39f02481"
server_key_iv: "5dc2972c290338c704ad89d7d0b42e8d"
server_key_version: "encryptionkey@0"
I20260928 06:33:49.944077 24615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:49.988714 24615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:49.991241 24615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:49.991394 24615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:49.992064 24615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:33:49.992112 24615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:49.992142 24615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:33:49.992161 24615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 06:33:49 dist-test-slave-rrj5 krb5kdc[24497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577229, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.157.65@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM
I20260928 06:33:50.000118 24615 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.65:37273
I20260928 06:33:50.000177 24746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.65:37273 every 8 connection(s)
I20260928 06:33:50.000483 24615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
I20260928 06:33:50.003504 24593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:49.992785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:39351 (local address 127.16.157.126:44663)
0928 06:33:49.992837 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:49.992840 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:49.992901 (+ 61us) server_negotiation.cc:407] Connection header received
0928 06:33:49.993665 (+ 764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:49.993667 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:49.993682 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:49.993714 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:49.994684 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:49.995495 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:49.997005 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:49.997359 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:49.997401 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:50.000171 (+ 2770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:50.000173 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:50.000174 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:50.000183 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:50.002060 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:50.002549 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:50.002550 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:50.002551 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:50.002614 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:50.002855 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:50.002855 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:50.002856 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:50.002994 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:50.003066 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:50.003307 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 06:33:50.003409 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:33:50.003589 24639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:49.992496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:44663 (local address 127.16.157.65:39351)
0928 06:33:49.992775 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:49.992817 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:49.993500 (+ 683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:49.993901 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:49.993905 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:49.994271 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:49.994565 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:49.994572 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:49.995621 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:49.995622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:49.996899 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:49.996901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:49.997506 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:49.997507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:49.997666 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:49.998310 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:49.998320 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:50.000060 (+ 1740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:50.002170 (+ 2110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:50.002179 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:50.002181 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:50.002462 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:50.002683 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:50.002684 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:50.002684 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:50.002787 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:50.003060 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:50.003064 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:50.003175 (+ 111us) client_negotiation.cc:769] Sending connection context
0928 06:33:50.003255 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 06:33:50.003408 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":81,"threads_started":1}
I20260928 06:33:50.004321 24747 heartbeater.cc:347] Connected to a master server at 127.16.157.126:44663
I20260928 06:33:50.004431 24747 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:50.004618 24747 heartbeater.cc:511] Master 127.16.157.126:44663 requested a full tablet report, sending...
I20260928 06:33:50.005436 24542 ts_manager.cc:194] Registered new tserver with Master: 32392e70db0445fe9341ff89e046a25d (127.16.157.65:37273)
I20260928 06:33:50.005759 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 24615
I20260928 06:33:50.005903 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/ts-0/wal/instance
I20260928 06:33:50.006088 17013 external_mini_cluster.cc:1428] Setting key dae4ee534349477e27e9ec2113da0eab
I20260928 06:33:50.006385 24542 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.157.65@KRBTEST.COM'} at 127.16.157.65:39351
I20260928 06:33:50.012830 24593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:50.005369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:48789 (local address 127.16.157.126:44663)
0928 06:33:50.006585 (+ 1216us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:50.006587 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:50.006596 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:33:50.006613 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:50.006614 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:50.006623 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:50.006679 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:50.007100 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.007882 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.008952 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.009134 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.009192 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:50.010177 (+ 985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:50.010178 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:50.010178 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:50.010185 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:50.011675 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:50.012043 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:50.012044 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:50.012044 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:50.012102 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:50.012321 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:50.012322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:50.012322 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:50.012429 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:50.012500 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:50.012658 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 06:33:50.012732 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1198}
I20260928 06:33:50.012840 24753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:50.005326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:44663 (local address 127.16.157.65:48789)
0928 06:33:50.005468 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:50.005478 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:50.005529 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:50.006723 (+ 1194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:50.006725 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:50.006939 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:50.007008 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:50.007010 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.008018 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.008019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:50.008863 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:50.008866 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.009214 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.009215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:50.009289 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:50.009790 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:50.009795 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:50.010069 (+ 274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:50.011780 (+ 1711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:50.011781 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:50.011781 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:50.011957 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:50.012175 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:50.012176 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:50.012176 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:50.012241 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:50.012504 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:50.012505 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:50.012549 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 06:33:50.012633 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 06:33:50.012732 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":89,"threads_started":1}
WARNING: no policy specified for oryx/127.16.157.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.157.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.157.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/krb5kdc/oryx_127.16.157.66.keytab.
Entry for principal oryx/127.16.157.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/krb5kdc/oryx_127.16.157.66.keytab.
I20260928 06:33:50.034665 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.66:0
--local_ip_for_outbound_sockets=127.16.157.66
--webserver_interface=127.16.157.66
--webserver_port=0
--tserver_master_addrs=127.16.157.126:44663
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/krb5kdc/oryx_127.16.157.66.keytab
--principal=oryx/127.16.157.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:37963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:50.118561 24756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:50.118781 24756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:50.118820 24756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:50.120009 24756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:50.120074 24756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:50.120107 24756 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:33:50.120148 24756 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.66
I20260928 06:33:50.121701 24756 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:37963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/krb5kdc/oryx_127.16.157.66.keytab
--principal=oryx/127.16.157.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.157.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.157.126:44663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24756
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.66
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:50.121953 24756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:50.122174 24756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:50.127061 24764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:50.127100 24762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:50.127105 24761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:50.128060 24756 server_base.cc:1034] running on GCE node
I20260928 06:33:50.128289 24756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:50.128547 24756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:50.129694 24756 hybrid_clock.cc:648] HybridClock initialized: now 1790577230129681 us; error 27 us; skew 500 ppm
Sep 28 06:33:50 dist-test-slave-rrj5 krb5kdc[24497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577230, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.157.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:50.132532 24756 init.cc:377] Logged in from keytab as oryx/127.16.157.66@KRBTEST.COM (short username oryx)
I20260928 06:33:50.133049 24756 webserver.cc:460] Webserver started at http://127.16.157.66:44185/ using document root <none> and password file <none>
I20260928 06:33:50.133266 24756 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:50.133342 24756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:50.133494 24756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:50.134630 24756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/ts-1/data/instance:
uuid: "524dc62a7f9740b5b29c93bab94dafeb"
format_stamp: "Formatted at 2026-09-28 06:33:50 on dist-test-slave-rrj5"
server_key: "31e7015aecfc077ddf6944ab6ed9b238"
server_key_iv: "d013ecc0c3a7d6016207941b4e1baebf"
server_key_version: "encryptionkey@0"
I20260928 06:33:50.134970 24756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/ts-1/wal/instance:
uuid: "524dc62a7f9740b5b29c93bab94dafeb"
format_stamp: "Formatted at 2026-09-28 06:33:50 on dist-test-slave-rrj5"
server_key: "31e7015aecfc077ddf6944ab6ed9b238"
server_key_iv: "d013ecc0c3a7d6016207941b4e1baebf"
server_key_version: "encryptionkey@0"
I20260928 06:33:50.136353 24756 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:33:50.137105 24771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:50.137296 24756 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:50.137398 24756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/ts-1/wal
uuid: "524dc62a7f9740b5b29c93bab94dafeb"
format_stamp: "Formatted at 2026-09-28 06:33:50 on dist-test-slave-rrj5"
server_key: "31e7015aecfc077ddf6944ab6ed9b238"
server_key_iv: "d013ecc0c3a7d6016207941b4e1baebf"
server_key_version: "encryptionkey@0"
I20260928 06:33:50.137475 24756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:50.162092 24756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:50.164353 24756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:50.164515 24756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:50.165113 24756 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:33:50.165174 24756 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:50.165207 24756 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:33:50.165230 24756 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 06:33:50 dist-test-slave-rrj5 krb5kdc[24497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577230, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.157.66@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM
I20260928 06:33:50.172704 24756 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.66:32923
I20260928 06:33:50.173076 24756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
I20260928 06:33:50.173524 24886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.66:32923 every 8 connection(s)
I20260928 06:33:50.175945 24593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:50.165444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.66:54923 (local address 127.16.157.126:44663)
0928 06:33:50.165488 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:50.165490 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:50.165771 (+ 281us) server_negotiation.cc:407] Connection header received
0928 06:33:50.166428 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:50.166430 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:50.166443 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:50.166529 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:50.167275 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.168114 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.169676 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.170056 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.170143 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:50.172892 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:50.172893 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:50.172894 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:50.172902 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:50.174636 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:50.175025 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:50.175026 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:50.175027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:50.175090 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:50.175333 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:50.175334 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:50.175335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:50.175477 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:50.175551 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:50.175776 (+ 225us) server_negotiation.cc:299] Negotiation successful
0928 06:33:50.175855 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 06:33:50.176011 24780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:50.165447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:44663 (local address 127.16.157.66:54923)
0928 06:33:50.165609 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:50.165656 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:50.166312 (+ 656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:50.166601 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:50.166605 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:50.166935 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:50.167184 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:50.167190 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.168223 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.168224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:50.169501 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:50.169504 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.170197 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.170199 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:50.170337 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:50.170920 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:50.170931 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:50.172807 (+ 1876us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:50.174734 (+ 1927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:50.174739 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:50.174741 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:50.174945 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:50.175161 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:50.175161 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:50.175162 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:50.175261 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:50.175544 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:50.175548 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:50.175646 (+ 98us) client_negotiation.cc:769] Sending connection context
0928 06:33:50.175735 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 06:33:50.175843 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":69,"threads_started":1}
I20260928 06:33:50.176539 24887 heartbeater.cc:347] Connected to a master server at 127.16.157.126:44663
I20260928 06:33:50.176621 24887 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:50.176789 24887 heartbeater.cc:511] Master 127.16.157.126:44663 requested a full tablet report, sending...
I20260928 06:33:50.177193 24542 ts_manager.cc:194] Registered new tserver with Master: 524dc62a7f9740b5b29c93bab94dafeb (127.16.157.66:32923)
I20260928 06:33:50.177647 24542 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.157.66@KRBTEST.COM'} at 127.16.157.66:54923
I20260928 06:33:50.179937 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 24756
I20260928 06:33:50.180056 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/ts-1/wal/instance
I20260928 06:33:50.180248 17013 external_mini_cluster.cc:1428] Setting key 1bcd2b70c6d62d57f5436e8144f39812
I20260928 06:33:50.183770 24593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:50.177683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.66:36637 (local address 127.16.157.126:44663)
0928 06:33:50.177788 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:50.177790 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:50.177927 (+ 137us) server_negotiation.cc:407] Connection header received
0928 06:33:50.177970 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:50.177971 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:50.177981 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:50.178039 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:50.178388 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.179151 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.179841 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.180037 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.180119 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:50.181274 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:50.181275 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:50.181276 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:50.181282 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:50.182642 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:50.182966 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:50.182967 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:50.182967 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:50.183027 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:50.183246 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:50.183247 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:50.183247 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:50.183385 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:50.183463 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:50.183613 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 06:33:50.183684 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 06:33:50.183797 24893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:50.177529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:44663 (local address 127.16.157.66:36637)
0928 06:33:50.177854 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:50.177865 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:50.177918 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:50.178042 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:50.178043 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:50.178238 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:50.178309 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:50.178311 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.179252 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.179253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:50.179743 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:50.179745 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.180121 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.180122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:50.180186 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:50.180839 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:50.180843 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:50.181175 (+ 332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:50.182730 (+ 1555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:50.182731 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:50.182731 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:50.182894 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:50.183111 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:50.183112 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:50.183113 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:50.183169 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:50.183454 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:50.183455 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:50.183499 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 06:33:50.183589 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 06:33:50.183686 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":101,"threads_started":1}
WARNING: no policy specified for oryx/127.16.157.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.157.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.157.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/krb5kdc/oryx_127.16.157.67.keytab.
Entry for principal oryx/127.16.157.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/krb5kdc/oryx_127.16.157.67.keytab.
I20260928 06:33:50.210337 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.67:0
--local_ip_for_outbound_sockets=127.16.157.67
--webserver_interface=127.16.157.67
--webserver_port=0
--tserver_master_addrs=127.16.157.126:44663
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/krb5kdc/oryx_127.16.157.67.keytab
--principal=oryx/127.16.157.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:37963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:50.325410 24896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:50.325666 24896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:50.325698 24896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:50.326861 24896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:50.326900 24896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:50.326920 24896 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:33:50.326937 24896 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.67
I20260928 06:33:50.328533 24896 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:37963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/krb5kdc/oryx_127.16.157.67.keytab
--principal=oryx/127.16.157.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.157.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.157.126:44663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24896
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.67
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:50.328783 24896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:50.329010 24896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:50.333894 24901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:50.334014 24896 server_base.cc:1034] running on GCE node
W20260928 06:33:50.333904 24904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:50.334127 24902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:50.334367 24896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:50.334607 24896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:50.335785 24896 hybrid_clock.cc:648] HybridClock initialized: now 1790577230335771 us; error 33 us; skew 500 ppm
Sep 28 06:33:50 dist-test-slave-rrj5 krb5kdc[24497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577230, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.157.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:50.338471 24896 init.cc:377] Logged in from keytab as oryx/127.16.157.67@KRBTEST.COM (short username oryx)
I20260928 06:33:50.338995 24896 webserver.cc:460] Webserver started at http://127.16.157.67:45981/ using document root <none> and password file <none>
I20260928 06:33:50.339275 24896 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:50.339339 24896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:50.339468 24896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:50.340838 24896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/ts-2/data/instance:
uuid: "1940c10288ad429e8bee620822bcd4e5"
format_stamp: "Formatted at 2026-09-28 06:33:50 on dist-test-slave-rrj5"
server_key: "bd5c9cb9a141c60f1ecbbf2d89ccc0d0"
server_key_iv: "9d4042dc6759b4fc8ad67053b977a3a3"
server_key_version: "encryptionkey@0"
I20260928 06:33:50.341236 24896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/ts-2/wal/instance:
uuid: "1940c10288ad429e8bee620822bcd4e5"
format_stamp: "Formatted at 2026-09-28 06:33:50 on dist-test-slave-rrj5"
server_key: "bd5c9cb9a141c60f1ecbbf2d89ccc0d0"
server_key_iv: "9d4042dc6759b4fc8ad67053b977a3a3"
server_key_version: "encryptionkey@0"
I20260928 06:33:50.342952 24896 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260928 06:33:50.343813 24911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:50.343995 24896 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:33:50.344097 24896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/ts-2/wal
uuid: "1940c10288ad429e8bee620822bcd4e5"
format_stamp: "Formatted at 2026-09-28 06:33:50 on dist-test-slave-rrj5"
server_key: "bd5c9cb9a141c60f1ecbbf2d89ccc0d0"
server_key_iv: "9d4042dc6759b4fc8ad67053b977a3a3"
server_key_version: "encryptionkey@0"
I20260928 06:33:50.344179 24896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:50.381601 24896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:50.384078 24896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:50.384230 24896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:50.384898 24896 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:33:50.384953 24896 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:50.384984 24896 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:33:50.385025 24896 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 06:33:50 dist-test-slave-rrj5 krb5kdc[24497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577230, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.157.67@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM
I20260928 06:33:50.392259 24896 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.67:39535
I20260928 06:33:50.392671 24896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
I20260928 06:33:50.392815 25028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.67:39535 every 8 connection(s)
I20260928 06:33:50.395681 24593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:50.385653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:32843 (local address 127.16.157.126:44663)
0928 06:33:50.385717 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:50.385720 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:50.385734 (+ 14us) server_negotiation.cc:407] Connection header received
0928 06:33:50.386230 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:50.386232 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:50.386245 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:50.386304 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:50.387147 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.387981 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.389531 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.389882 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.389959 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:50.392637 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:50.392639 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:50.392639 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:50.392648 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:50.394276 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:50.394693 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:50.394694 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:50.394695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:50.394758 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:50.395023 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:50.395024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:50.395025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:50.395145 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:50.395217 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:50.395511 (+ 294us) server_negotiation.cc:299] Negotiation successful
0928 06:33:50.395593 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 06:33:50.395879 24920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:50.385303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:44663 (local address 127.16.157.67:32843)
0928 06:33:50.385472 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:50.385514 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:50.386110 (+ 596us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:50.386387 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:50.386390 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:50.386759 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:50.387050 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:50.387056 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.388067 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.388069 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:50.389443 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:50.389446 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.389985 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.389987 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:50.390150 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:50.390749 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:50.390765 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:50.392536 (+ 1771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:50.394373 (+ 1837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:50.394379 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:50.394381 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:50.394609 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:50.394827 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:50.394828 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:50.394828 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:50.394946 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:50.395219 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:50.395226 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:50.395364 (+ 138us) client_negotiation.cc:769] Sending connection context
0928 06:33:50.395476 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 06:33:50.395635 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":95,"threads_started":1}
I20260928 06:33:50.396001 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 24896
I20260928 06:33:50.396111 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/ts-2/wal/instance
I20260928 06:33:50.396296 17013 external_mini_cluster.cc:1428] Setting key 9776b6938b6bec2534e19507a3e6eafa
I20260928 06:33:50.397364 25029 heartbeater.cc:347] Connected to a master server at 127.16.157.126:44663
I20260928 06:33:50.397461 25029 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:50.397655 25029 heartbeater.cc:511] Master 127.16.157.126:44663 requested a full tablet report, sending...
I20260928 06:33:50.398036 24542 ts_manager.cc:194] Registered new tserver with Master: 1940c10288ad429e8bee620822bcd4e5 (127.16.157.67:39535)
I20260928 06:33:50.398487 24542 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.157.67@KRBTEST.COM'} at 127.16.157.67:32843
I20260928 06:33:50.403693 24593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:50.397525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:44297 (local address 127.16.157.126:44663)
0928 06:33:50.397593 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:50.397595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:50.397858 (+ 263us) server_negotiation.cc:407] Connection header received
0928 06:33:50.397959 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:50.397960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:50.397971 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:50.398023 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:50.398391 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.399062 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.400100 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.400278 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.400327 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:50.401244 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:50.401245 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:50.401246 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:50.401252 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:50.402632 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:50.403012 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:50.403013 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:50.403013 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:50.403059 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:50.403246 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:50.403246 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:50.403247 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:50.403336 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:50.403385 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:50.403540 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 06:33:50.403638 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:33:50.403795 25035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:50.397593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:44663 (local address 127.16.157.67:44297)
0928 06:33:50.397788 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:50.397800 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:50.397885 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:50.398031 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:50.398032 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:50.398237 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:50.398307 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:50.398309 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.399170 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.399171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:50.400010 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:50.400012 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.400349 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.400349 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:50.400419 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:50.400801 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:50.400806 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:50.401163 (+ 357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:50.402716 (+ 1553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:50.402717 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:50.402718 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:50.402939 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:50.403123 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:50.403124 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:50.403124 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:50.403189 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:50.403399 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:50.403399 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:50.403447 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 06:33:50.403526 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 06:33:50.403695 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":77,"threads_started":1}
I20260928 06:33:50.408006 17013 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 06:33:50 dist-test-slave-rrj5 krb5kdc[24497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577230, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 06:33:50 dist-test-slave-rrj5 krb5kdc[24497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:33:50 dist-test-slave-rrj5 krb5kdc[24497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 06:33:50.430203 24593 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:33:50.424391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51536 (local address 127.16.157.126:44663)
0928 06:33:50.424459 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:50.424461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:50.424471 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:33:50.424500 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:50.424501 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:50.424511 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:50.424582 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:50.425115 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.425950 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.426691 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.427019 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.427095 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:50.430123 (+ 3028us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51536: BlockingRecv error: recv got EOF from 127.0.0.1:51536 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 28 06:33:50 dist-test-slave-rrj5 krb5kdc[24497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577230, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM
I20260928 06:33:50.438239 24593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:50.430954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51548 (local address 127.16.157.126:44663)
0928 06:33:50.430992 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:50.430994 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:50.431122 (+ 128us) server_negotiation.cc:407] Connection header received
0928 06:33:50.431165 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:50.431166 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:50.431176 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:50.431231 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:50.431664 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.432379 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.433029 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.433323 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.433397 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:50.435587 (+ 2190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:50.435589 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:50.435589 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:50.435596 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:50.437089 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:50.437406 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:50.437407 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:50.437408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:50.437469 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:50.437756 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:50.437757 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:50.437757 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:50.437896 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:50.437998 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:50.438074 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 06:33:50.438148 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 06:33:50.440097 24542 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51548:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:33:50.442286 24542 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:33:50.449659 25053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:50.444720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.65:37273 (local address 127.0.0.1:34288)
0928 06:33:50.445101 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:50.445112 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:50.445170 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:50.445510 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:50.445513 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:50.445515 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:33:50.445601 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:50.445602 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.446893 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.446895 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:50.448306 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:50.448309 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.449354 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.449355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:50.449444 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:50.449456 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 06:33:50.449517 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 06:33:50.449563 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":2,"thread_start_us":101,"threads_started":1}
I20260928 06:33:50.449934 25051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:50.444720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.67:39535 (local address 127.0.0.1:38864)
0928 06:33:50.445537 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:50.445547 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:50.445610 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:50.446197 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:50.446198 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:50.446200 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:33:50.446234 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:50.446235 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.447459 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.447460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:50.448629 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:50.448632 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.449692 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.449694 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:50.449766 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:50.449773 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 06:33:50.449807 (+ 34us) client_negotiation.cc:240] Negotiation successful
0928 06:33:50.449860 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":804,"mutex_wait_us":2,"thread_start_us":97,"threads_started":1}
I20260928 06:33:50.449957 25054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:50.444960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34288 (local address 127.16.157.65:37273)
0928 06:33:50.445263 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:50.445267 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:50.445275 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:33:50.445311 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:50.445312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:50.445325 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:50.445432 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:33:50.445681 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.446783 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.448399 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.449191 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.449255 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:50.449321 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:50.449816 (+ 495us) server_negotiation.cc:299] Negotiation successful
0928 06:33:50.449872 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":50,"threads_started":1}
I20260928 06:33:50.450685 25055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:50.444946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42992 (local address 127.16.157.66:32923)
0928 06:33:50.445730 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:50.445735 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:50.445744 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:33:50.445783 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:50.445785 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:50.445799 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:50.445900 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:33:50.446059 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.447163 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.448867 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.450213 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.450261 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:50.450338 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:50.450546 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 06:33:50.450588 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":746,"thread_start_us":93,"threads_started":1}
I20260928 06:33:50.450738 24680 tablet_service.cc:1467] Processing CreateTablet for tablet 06706edea8344ada8f0879fb044c9f2f (DEFAULT_TABLE table=test-table [id=a65ea94fe8fd4955bc3845b392b5bffb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:33:50.450960 25052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:50.444715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.66:32923 (local address 127.0.0.1:42992)
0928 06:33:50.444881 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:50.444923 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:50.444992 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:50.445898 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:50.445899 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:50.445901 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:33:50.445963 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:50.445965 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.447270 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.447271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:50.448738 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:50.448740 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.450397 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.450397 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:50.450459 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:50.450463 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:33:50.450841 (+ 378us) client_negotiation.cc:240] Negotiation successful
0928 06:33:50.450881 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":3,"thread_start_us":36,"threads_started":1}
I20260928 06:33:50.451133 24680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06706edea8344ada8f0879fb044c9f2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:50.451617 24812 tablet_service.cc:1467] Processing CreateTablet for tablet 06706edea8344ada8f0879fb044c9f2f (DEFAULT_TABLE table=test-table [id=a65ea94fe8fd4955bc3845b392b5bffb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:33:50.451929 24812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06706edea8344ada8f0879fb044c9f2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:50.453965 25058 tablet_bootstrap.cc:492] T 06706edea8344ada8f0879fb044c9f2f P 32392e70db0445fe9341ff89e046a25d: Bootstrap starting.
I20260928 06:33:50.454514 25056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:50.444968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38864 (local address 127.16.157.67:39535)
0928 06:33:50.446027 (+ 1059us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:50.446031 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:50.446041 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:33:50.446081 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:50.446083 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:50.446100 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:50.446303 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:33:50.446362 (+ 59us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.447355 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.448807 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.449537 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.454207 (+ 4670us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:50.454300 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:50.454330 (+ 30us) server_negotiation.cc:299] Negotiation successful
0928 06:33:50.454389 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1021,"thread_start_us":101,"threads_started":1}
I20260928 06:33:50.455152 25058 tablet_bootstrap.cc:654] T 06706edea8344ada8f0879fb044c9f2f P 32392e70db0445fe9341ff89e046a25d: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:50.455090 24962 tablet_service.cc:1467] Processing CreateTablet for tablet 06706edea8344ada8f0879fb044c9f2f (DEFAULT_TABLE table=test-table [id=a65ea94fe8fd4955bc3845b392b5bffb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:33:50.455444 24962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06706edea8344ada8f0879fb044c9f2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:50.455432 25059 tablet_bootstrap.cc:492] T 06706edea8344ada8f0879fb044c9f2f P 524dc62a7f9740b5b29c93bab94dafeb: Bootstrap starting.
I20260928 06:33:50.455638 25058 log.cc:824] T 06706edea8344ada8f0879fb044c9f2f P 32392e70db0445fe9341ff89e046a25d: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:50.456050 25059 tablet_bootstrap.cc:654] T 06706edea8344ada8f0879fb044c9f2f P 524dc62a7f9740b5b29c93bab94dafeb: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:50.456388 25059 log.cc:824] T 06706edea8344ada8f0879fb044c9f2f P 524dc62a7f9740b5b29c93bab94dafeb: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:50.456470 25058 tablet_bootstrap.cc:492] T 06706edea8344ada8f0879fb044c9f2f P 32392e70db0445fe9341ff89e046a25d: No bootstrap required, opened a new log
I20260928 06:33:50.456522 25058 ts_tablet_manager.cc:1397] T 06706edea8344ada8f0879fb044c9f2f P 32392e70db0445fe9341ff89e046a25d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 06:33:50.458402 25058 raft_consensus.cc:348] T 06706edea8344ada8f0879fb044c9f2f P 32392e70db0445fe9341ff89e046a25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1940c10288ad429e8bee620822bcd4e5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39535 } } peers { permanent_uuid: "524dc62a7f9740b5b29c93bab94dafeb" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32923 } } peers { permanent_uuid: "32392e70db0445fe9341ff89e046a25d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 37273 } }
I20260928 06:33:50.458568 25058 raft_consensus.cc:374] T 06706edea8344ada8f0879fb044c9f2f P 32392e70db0445fe9341ff89e046a25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:50.458606 25058 raft_consensus.cc:729] T 06706edea8344ada8f0879fb044c9f2f P 32392e70db0445fe9341ff89e046a25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32392e70db0445fe9341ff89e046a25d, State: Initialized, Role: FOLLOWER
I20260928 06:33:50.458738 25058 consensus_queue.cc:260] T 06706edea8344ada8f0879fb044c9f2f P 32392e70db0445fe9341ff89e046a25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1940c10288ad429e8bee620822bcd4e5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39535 } } peers { permanent_uuid: "524dc62a7f9740b5b29c93bab94dafeb" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32923 } } peers { permanent_uuid: "32392e70db0445fe9341ff89e046a25d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 37273 } }
I20260928 06:33:50.458743 25062 tablet_bootstrap.cc:492] T 06706edea8344ada8f0879fb044c9f2f P 1940c10288ad429e8bee620822bcd4e5: Bootstrap starting.
I20260928 06:33:50.459007 25058 ts_tablet_manager.cc:1428] T 06706edea8344ada8f0879fb044c9f2f P 32392e70db0445fe9341ff89e046a25d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 06:33:50.459062 24747 heartbeater.cc:503] Master 127.16.157.126:44663 was elected leader, sending a full tablet report...
I20260928 06:33:50.459494 25062 tablet_bootstrap.cc:654] T 06706edea8344ada8f0879fb044c9f2f P 1940c10288ad429e8bee620822bcd4e5: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:50.459600 25059 tablet_bootstrap.cc:492] T 06706edea8344ada8f0879fb044c9f2f P 524dc62a7f9740b5b29c93bab94dafeb: No bootstrap required, opened a new log
I20260928 06:33:50.459671 25059 ts_tablet_manager.cc:1397] T 06706edea8344ada8f0879fb044c9f2f P 524dc62a7f9740b5b29c93bab94dafeb: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260928 06:33:50.459856 25062 log.cc:824] T 06706edea8344ada8f0879fb044c9f2f P 1940c10288ad429e8bee620822bcd4e5: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:50.460660 25062 tablet_bootstrap.cc:492] T 06706edea8344ada8f0879fb044c9f2f P 1940c10288ad429e8bee620822bcd4e5: No bootstrap required, opened a new log
I20260928 06:33:50.460713 25062 ts_tablet_manager.cc:1397] T 06706edea8344ada8f0879fb044c9f2f P 1940c10288ad429e8bee620822bcd4e5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:33:50.461315 25059 raft_consensus.cc:348] T 06706edea8344ada8f0879fb044c9f2f P 524dc62a7f9740b5b29c93bab94dafeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1940c10288ad429e8bee620822bcd4e5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39535 } } peers { permanent_uuid: "524dc62a7f9740b5b29c93bab94dafeb" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32923 } } peers { permanent_uuid: "32392e70db0445fe9341ff89e046a25d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 37273 } }
I20260928 06:33:50.461464 25059 raft_consensus.cc:374] T 06706edea8344ada8f0879fb044c9f2f P 524dc62a7f9740b5b29c93bab94dafeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:50.461511 25059 raft_consensus.cc:729] T 06706edea8344ada8f0879fb044c9f2f P 524dc62a7f9740b5b29c93bab94dafeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 524dc62a7f9740b5b29c93bab94dafeb, State: Initialized, Role: FOLLOWER
I20260928 06:33:50.461637 25059 consensus_queue.cc:260] T 06706edea8344ada8f0879fb044c9f2f P 524dc62a7f9740b5b29c93bab94dafeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1940c10288ad429e8bee620822bcd4e5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39535 } } peers { permanent_uuid: "524dc62a7f9740b5b29c93bab94dafeb" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32923 } } peers { permanent_uuid: "32392e70db0445fe9341ff89e046a25d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 37273 } }
I20260928 06:33:50.461889 25059 ts_tablet_manager.cc:1428] T 06706edea8344ada8f0879fb044c9f2f P 524dc62a7f9740b5b29c93bab94dafeb: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 06:33:50.461972 24887 heartbeater.cc:503] Master 127.16.157.126:44663 was elected leader, sending a full tablet report...
I20260928 06:33:50.462270 25062 raft_consensus.cc:348] T 06706edea8344ada8f0879fb044c9f2f P 1940c10288ad429e8bee620822bcd4e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1940c10288ad429e8bee620822bcd4e5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39535 } } peers { permanent_uuid: "524dc62a7f9740b5b29c93bab94dafeb" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32923 } } peers { permanent_uuid: "32392e70db0445fe9341ff89e046a25d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 37273 } }
I20260928 06:33:50.462411 25062 raft_consensus.cc:374] T 06706edea8344ada8f0879fb044c9f2f P 1940c10288ad429e8bee620822bcd4e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:50.462455 25062 raft_consensus.cc:729] T 06706edea8344ada8f0879fb044c9f2f P 1940c10288ad429e8bee620822bcd4e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1940c10288ad429e8bee620822bcd4e5, State: Initialized, Role: FOLLOWER
I20260928 06:33:50.462566 25062 consensus_queue.cc:260] T 06706edea8344ada8f0879fb044c9f2f P 1940c10288ad429e8bee620822bcd4e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1940c10288ad429e8bee620822bcd4e5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39535 } } peers { permanent_uuid: "524dc62a7f9740b5b29c93bab94dafeb" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32923 } } peers { permanent_uuid: "32392e70db0445fe9341ff89e046a25d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 37273 } }
I20260928 06:33:50.462833 25062 ts_tablet_manager.cc:1428] T 06706edea8344ada8f0879fb044c9f2f P 1940c10288ad429e8bee620822bcd4e5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:33:50.462886 25029 heartbeater.cc:503] Master 127.16.157.126:44663 was elected leader, sending a full tablet report...
W20260928 06:33:50.501763 24748 tablet.cc:2369] T 06706edea8344ada8f0879fb044c9f2f P 32392e70db0445fe9341ff89e046a25d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:33:50.567289 25065 raft_consensus.cc:482] T 06706edea8344ada8f0879fb044c9f2f P 524dc62a7f9740b5b29c93bab94dafeb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:33:50.567422 25065 raft_consensus.cc:504] T 06706edea8344ada8f0879fb044c9f2f P 524dc62a7f9740b5b29c93bab94dafeb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1940c10288ad429e8bee620822bcd4e5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39535 } } peers { permanent_uuid: "524dc62a7f9740b5b29c93bab94dafeb" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32923 } } peers { permanent_uuid: "32392e70db0445fe9341ff89e046a25d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 37273 } }
I20260928 06:33:50.567843 25065 leader_election.cc:288] T 06706edea8344ada8f0879fb044c9f2f P 524dc62a7f9740b5b29c93bab94dafeb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1940c10288ad429e8bee620822bcd4e5 (127.16.157.67:39535), 32392e70db0445fe9341ff89e046a25d (127.16.157.65:37273)
I20260928 06:33:50.571389 24780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:50.568044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.67:39535 (local address 127.16.157.66:53963)
0928 06:33:50.568111 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:50.568122 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:50.568162 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:50.568403 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:50.568404 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:50.568406 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:33:50.568482 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:50.568483 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.569180 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.569181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:50.570310 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:50.570313 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.571096 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.571097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:50.571196 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:50.571200 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:33:50.571255 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 06:33:50.571297 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 06:33:50.571398 25056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:50.568198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.66:53963 (local address 127.16.157.67:39535)
0928 06:33:50.568247 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:50.568249 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:50.568261 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:33:50.568290 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:50.568291 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:50.568301 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:50.568354 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:33:50.568573 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.569093 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.570413 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.570980 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.571054 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:50.571063 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:50.571283 (+ 220us) server_negotiation.cc:299] Negotiation successful
0928 06:33:50.571335 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 06:33:50.571439 25054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:50.568247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.66:40107 (local address 127.16.157.65:37273)
0928 06:33:50.568291 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:50.568293 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:50.568303 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:33:50.568398 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:50.568400 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:50.568409 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:50.568439 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:33:50.568697 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.569420 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.570634 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.571052 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.571135 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:50.571143 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:50.571328 (+ 185us) server_negotiation.cc:299] Negotiation successful
0928 06:33:50.571366 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 06:33:50.571623 25067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:50.568058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.65:37273 (local address 127.16.157.66:40107)
0928 06:33:50.568245 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:50.568256 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:50.568308 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:50.568502 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:50.568503 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:50.568514 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:33:50.568612 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:50.568614 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.569522 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.569523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:50.570548 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:50.570550 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.571135 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.571136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:50.571210 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:50.571214 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:33:50.571510 (+ 296us) client_negotiation.cc:240] Negotiation successful
0928 06:33:50.571558 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":104,"threads_started":1}
I20260928 06:33:50.571934 24982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06706edea8344ada8f0879fb044c9f2f" candidate_uuid: "524dc62a7f9740b5b29c93bab94dafeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1940c10288ad429e8bee620822bcd4e5" is_pre_election: true
I20260928 06:33:50.571928 24701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06706edea8344ada8f0879fb044c9f2f" candidate_uuid: "524dc62a7f9740b5b29c93bab94dafeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32392e70db0445fe9341ff89e046a25d" is_pre_election: true
I20260928 06:33:50.572093 24701 raft_consensus.cc:2454] T 06706edea8344ada8f0879fb044c9f2f P 32392e70db0445fe9341ff89e046a25d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 524dc62a7f9740b5b29c93bab94dafeb in term 0.
I20260928 06:33:50.572093 24982 raft_consensus.cc:2454] T 06706edea8344ada8f0879fb044c9f2f P 1940c10288ad429e8bee620822bcd4e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 524dc62a7f9740b5b29c93bab94dafeb in term 0.
I20260928 06:33:50.572304 24774 leader_election.cc:302] T 06706edea8344ada8f0879fb044c9f2f P 524dc62a7f9740b5b29c93bab94dafeb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32392e70db0445fe9341ff89e046a25d, 524dc62a7f9740b5b29c93bab94dafeb; no voters:
I20260928 06:33:50.572504 25065 raft_consensus.cc:2781] T 06706edea8344ada8f0879fb044c9f2f P 524dc62a7f9740b5b29c93bab94dafeb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:33:50.572549 25065 raft_consensus.cc:482] T 06706edea8344ada8f0879fb044c9f2f P 524dc62a7f9740b5b29c93bab94dafeb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:33:50.572566 25065 raft_consensus.cc:3037] T 06706edea8344ada8f0879fb044c9f2f P 524dc62a7f9740b5b29c93bab94dafeb [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:50.573668 25065 raft_consensus.cc:504] T 06706edea8344ada8f0879fb044c9f2f P 524dc62a7f9740b5b29c93bab94dafeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1940c10288ad429e8bee620822bcd4e5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39535 } } peers { permanent_uuid: "524dc62a7f9740b5b29c93bab94dafeb" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32923 } } peers { permanent_uuid: "32392e70db0445fe9341ff89e046a25d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 37273 } }
I20260928 06:33:50.573801 25065 leader_election.cc:288] T 06706edea8344ada8f0879fb044c9f2f P 524dc62a7f9740b5b29c93bab94dafeb [CANDIDATE]: Term 1 election: Requested vote from peers 1940c10288ad429e8bee620822bcd4e5 (127.16.157.67:39535), 32392e70db0445fe9341ff89e046a25d (127.16.157.65:37273)
I20260928 06:33:50.573957 24982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06706edea8344ada8f0879fb044c9f2f" candidate_uuid: "524dc62a7f9740b5b29c93bab94dafeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1940c10288ad429e8bee620822bcd4e5"
I20260928 06:33:50.573979 24701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06706edea8344ada8f0879fb044c9f2f" candidate_uuid: "524dc62a7f9740b5b29c93bab94dafeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32392e70db0445fe9341ff89e046a25d"
I20260928 06:33:50.574034 24982 raft_consensus.cc:3037] T 06706edea8344ada8f0879fb044c9f2f P 1940c10288ad429e8bee620822bcd4e5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:50.574043 24701 raft_consensus.cc:3037] T 06706edea8344ada8f0879fb044c9f2f P 32392e70db0445fe9341ff89e046a25d [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:50.574855 24701 raft_consensus.cc:2454] T 06706edea8344ada8f0879fb044c9f2f P 32392e70db0445fe9341ff89e046a25d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 524dc62a7f9740b5b29c93bab94dafeb in term 1.
I20260928 06:33:50.574855 24982 raft_consensus.cc:2454] T 06706edea8344ada8f0879fb044c9f2f P 1940c10288ad429e8bee620822bcd4e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 524dc62a7f9740b5b29c93bab94dafeb in term 1.
I20260928 06:33:50.575032 24772 leader_election.cc:302] T 06706edea8344ada8f0879fb044c9f2f P 524dc62a7f9740b5b29c93bab94dafeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1940c10288ad429e8bee620822bcd4e5, 524dc62a7f9740b5b29c93bab94dafeb; no voters:
I20260928 06:33:50.575124 25065 raft_consensus.cc:2781] T 06706edea8344ada8f0879fb044c9f2f P 524dc62a7f9740b5b29c93bab94dafeb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:33:50.575259 25065 raft_consensus.cc:686] T 06706edea8344ada8f0879fb044c9f2f P 524dc62a7f9740b5b29c93bab94dafeb [term 1 LEADER]: Becoming Leader. State: Replica: 524dc62a7f9740b5b29c93bab94dafeb, State: Running, Role: LEADER
I20260928 06:33:50.575351 25065 consensus_queue.cc:237] T 06706edea8344ada8f0879fb044c9f2f P 524dc62a7f9740b5b29c93bab94dafeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1940c10288ad429e8bee620822bcd4e5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39535 } } peers { permanent_uuid: "524dc62a7f9740b5b29c93bab94dafeb" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32923 } } peers { permanent_uuid: "32392e70db0445fe9341ff89e046a25d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 37273 } }
I20260928 06:33:50.576205 24542 catalog_manager.cc:5495] T 06706edea8344ada8f0879fb044c9f2f P 524dc62a7f9740b5b29c93bab94dafeb reported cstate change: term changed from 0 to 1, leader changed from <none> to 524dc62a7f9740b5b29c93bab94dafeb (127.16.157.66). New cstate: current_term: 1 leader_uuid: "524dc62a7f9740b5b29c93bab94dafeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1940c10288ad429e8bee620822bcd4e5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "524dc62a7f9740b5b29c93bab94dafeb" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32392e70db0445fe9341ff89e046a25d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 37273 } health_report { overall_health: UNKNOWN } } }
I20260928 06:33:50.593031 25055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:50.590384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43006 (local address 127.16.157.66:32923)
0928 06:33:50.590448 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:50.590450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:50.590459 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:33:50.590495 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:50.590496 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:50.590516 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:50.590601 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:33:50.590735 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.591478 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.592192 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.592495 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.592567 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:50.592708 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:33:50.592837 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:33:50.592915 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:50.592937 (+ 22us) server_negotiation.cc:299] Negotiation successful
0928 06:33:50.592958 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 06:33:50.594965 24982 raft_consensus.cc:1261] T 06706edea8344ada8f0879fb044c9f2f P 1940c10288ad429e8bee620822bcd4e5 [term 1 FOLLOWER]: Refusing update from remote peer 524dc62a7f9740b5b29c93bab94dafeb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:33:50.594965 24701 raft_consensus.cc:1261] T 06706edea8344ada8f0879fb044c9f2f P 32392e70db0445fe9341ff89e046a25d [term 1 FOLLOWER]: Refusing update from remote peer 524dc62a7f9740b5b29c93bab94dafeb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:33:50.595305 25065 consensus_queue.cc:1035] T 06706edea8344ada8f0879fb044c9f2f P 524dc62a7f9740b5b29c93bab94dafeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "32392e70db0445fe9341ff89e046a25d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 37273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:33:50.595448 25068 consensus_queue.cc:1035] T 06706edea8344ada8f0879fb044c9f2f P 524dc62a7f9740b5b29c93bab94dafeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "1940c10288ad429e8bee620822bcd4e5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:33:50.598278 25071 mvcc.cc:205] Tried to move back new op lower bound from 7334204336513966080 to 7334204336437178368. Current Snapshot: MvccSnapshot[applied={T|T < 7334204336513966080}]
I20260928 06:33:50.598685 25072 mvcc.cc:205] Tried to move back new op lower bound from 7334204336513966080 to 7334204336437178368. Current Snapshot: MvccSnapshot[applied={T|T < 7334204336513966080}]
I20260928 06:33:50.601109 24542 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:33:50.601203 24542 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:33:50.601939 24542 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 217c529c34ff472987667c1af6f76a6b: Sending DeleteTablet for 3 replicas of tablet 06706edea8344ada8f0879fb044c9f2f
I20260928 06:33:50.602275 24812 tablet_service.cc:1514] Processing DeleteTablet for tablet 06706edea8344ada8f0879fb044c9f2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:33:50 UTC) from {username='oryx', principal='oryx/127.16.157.126@KRBTEST.COM'} at 127.0.0.1:42992
I20260928 06:33:50.602545 24680 tablet_service.cc:1514] Processing DeleteTablet for tablet 06706edea8344ada8f0879fb044c9f2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:33:50 UTC) from {username='oryx', principal='oryx/127.16.157.126@KRBTEST.COM'} at 127.0.0.1:34288
I20260928 06:33:50.602780 25081 tablet_replica.cc:330] T 06706edea8344ada8f0879fb044c9f2f P 32392e70db0445fe9341ff89e046a25d: stopping tablet replica
I20260928 06:33:50.602871 24962 tablet_service.cc:1514] Processing DeleteTablet for tablet 06706edea8344ada8f0879fb044c9f2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:33:50 UTC) from {username='oryx', principal='oryx/127.16.157.126@KRBTEST.COM'} at 127.0.0.1:38864
I20260928 06:33:50.602921 25081 raft_consensus.cc:2229] T 06706edea8344ada8f0879fb044c9f2f P 32392e70db0445fe9341ff89e046a25d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:33:50.603070 25081 raft_consensus.cc:2258] T 06706edea8344ada8f0879fb044c9f2f P 32392e70db0445fe9341ff89e046a25d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:33:50.603089 24542 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51548:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:33:50.603227 24542 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:33:50.603688 25081 ts_tablet_manager.cc:1905] T 06706edea8344ada8f0879fb044c9f2f P 32392e70db0445fe9341ff89e046a25d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:33:50.604584 24680 tablet_service.cc:1467] Processing CreateTablet for tablet 31587501e8ab4fe88dcfd5b87ee60256 (DEFAULT_TABLE table=test-table [id=ff733fa3c82f4ddfa814e02056ba27f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:33:50.604866 24962 tablet_service.cc:1467] Processing CreateTablet for tablet 31587501e8ab4fe88dcfd5b87ee60256 (DEFAULT_TABLE table=test-table [id=ff733fa3c82f4ddfa814e02056ba27f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:33:50.604995 24962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31587501e8ab4fe88dcfd5b87ee60256. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:50.605077 25082 tablet_replica.cc:330] T 06706edea8344ada8f0879fb044c9f2f P 1940c10288ad429e8bee620822bcd4e5: stopping tablet replica
I20260928 06:33:50.605234 25082 raft_consensus.cc:2229] T 06706edea8344ada8f0879fb044c9f2f P 1940c10288ad429e8bee620822bcd4e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:33:50.605383 25082 raft_consensus.cc:2258] T 06706edea8344ada8f0879fb044c9f2f P 1940c10288ad429e8bee620822bcd4e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:33:50.606710 25062 tablet_bootstrap.cc:492] T 31587501e8ab4fe88dcfd5b87ee60256 P 1940c10288ad429e8bee620822bcd4e5: Bootstrap starting.
I20260928 06:33:50.607142 25062 tablet_bootstrap.cc:654] T 31587501e8ab4fe88dcfd5b87ee60256 P 1940c10288ad429e8bee620822bcd4e5: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:50.607734 25062 tablet_bootstrap.cc:492] T 31587501e8ab4fe88dcfd5b87ee60256 P 1940c10288ad429e8bee620822bcd4e5: No bootstrap required, opened a new log
I20260928 06:33:50.607739 24812 tablet_service.cc:1467] Processing CreateTablet for tablet 31587501e8ab4fe88dcfd5b87ee60256 (DEFAULT_TABLE table=test-table [id=ff733fa3c82f4ddfa814e02056ba27f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:33:50.607777 25062 ts_tablet_manager.cc:1397] T 31587501e8ab4fe88dcfd5b87ee60256 P 1940c10288ad429e8bee620822bcd4e5: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:50.607843 24812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31587501e8ab4fe88dcfd5b87ee60256. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:50.607932 25062 raft_consensus.cc:348] T 31587501e8ab4fe88dcfd5b87ee60256 P 1940c10288ad429e8bee620822bcd4e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32392e70db0445fe9341ff89e046a25d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 37273 } } peers { permanent_uuid: "524dc62a7f9740b5b29c93bab94dafeb" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32923 } } peers { permanent_uuid: "1940c10288ad429e8bee620822bcd4e5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39535 } }
I20260928 06:33:50.608002 25062 raft_consensus.cc:374] T 31587501e8ab4fe88dcfd5b87ee60256 P 1940c10288ad429e8bee620822bcd4e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:50.608021 25062 raft_consensus.cc:729] T 31587501e8ab4fe88dcfd5b87ee60256 P 1940c10288ad429e8bee620822bcd4e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1940c10288ad429e8bee620822bcd4e5, State: Initialized, Role: FOLLOWER
I20260928 06:33:50.608067 25062 consensus_queue.cc:260] T 31587501e8ab4fe88dcfd5b87ee60256 P 1940c10288ad429e8bee620822bcd4e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32392e70db0445fe9341ff89e046a25d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 37273 } } peers { permanent_uuid: "524dc62a7f9740b5b29c93bab94dafeb" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32923 } } peers { permanent_uuid: "1940c10288ad429e8bee620822bcd4e5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39535 } }
I20260928 06:33:50.608143 25062 ts_tablet_manager.cc:1428] T 31587501e8ab4fe88dcfd5b87ee60256 P 1940c10288ad429e8bee620822bcd4e5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:50.609194 25059 tablet_bootstrap.cc:492] T 31587501e8ab4fe88dcfd5b87ee60256 P 524dc62a7f9740b5b29c93bab94dafeb: Bootstrap starting.
I20260928 06:33:50.609477 24680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31587501e8ab4fe88dcfd5b87ee60256. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:50.609730 25059 tablet_bootstrap.cc:654] T 31587501e8ab4fe88dcfd5b87ee60256 P 524dc62a7f9740b5b29c93bab94dafeb: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:50.610812 25058 tablet_bootstrap.cc:492] T 31587501e8ab4fe88dcfd5b87ee60256 P 32392e70db0445fe9341ff89e046a25d: Bootstrap starting.
I20260928 06:33:50.611210 25058 tablet_bootstrap.cc:654] T 31587501e8ab4fe88dcfd5b87ee60256 P 32392e70db0445fe9341ff89e046a25d: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:50.612205 25081 ts_tablet_manager.cc:1918] T 06706edea8344ada8f0879fb044c9f2f P 32392e70db0445fe9341ff89e046a25d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:33:50.612282 25081 log.cc:1196] T 06706edea8344ada8f0879fb044c9f2f P 32392e70db0445fe9341ff89e046a25d: Deleting WAL directory at /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/ts-0/wal/wals/06706edea8344ada8f0879fb044c9f2f
I20260928 06:33:50.612532 25081 ts_tablet_manager.cc:1939] T 06706edea8344ada8f0879fb044c9f2f P 32392e70db0445fe9341ff89e046a25d: Deleting consensus metadata
I20260928 06:33:50.612932 25082 ts_tablet_manager.cc:1905] T 06706edea8344ada8f0879fb044c9f2f P 1940c10288ad429e8bee620822bcd4e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:33:50.613224 24528 catalog_manager.cc:4841] TS 32392e70db0445fe9341ff89e046a25d (127.16.157.65:37273): tablet 06706edea8344ada8f0879fb044c9f2f (table test-table [id=a65ea94fe8fd4955bc3845b392b5bffb]) successfully deleted
I20260928 06:33:50.613555 25080 tablet_replica.cc:330] T 06706edea8344ada8f0879fb044c9f2f P 524dc62a7f9740b5b29c93bab94dafeb: stopping tablet replica
I20260928 06:33:50.613682 25080 raft_consensus.cc:2229] T 06706edea8344ada8f0879fb044c9f2f P 524dc62a7f9740b5b29c93bab94dafeb [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:33:50.613797 25080 raft_consensus.cc:2258] T 06706edea8344ada8f0879fb044c9f2f P 524dc62a7f9740b5b29c93bab94dafeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:33:50.614207 25059 tablet_bootstrap.cc:492] T 31587501e8ab4fe88dcfd5b87ee60256 P 524dc62a7f9740b5b29c93bab94dafeb: No bootstrap required, opened a new log
I20260928 06:33:50.614259 25059 ts_tablet_manager.cc:1397] T 31587501e8ab4fe88dcfd5b87ee60256 P 524dc62a7f9740b5b29c93bab94dafeb: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260928 06:33:50.614391 25059 raft_consensus.cc:348] T 31587501e8ab4fe88dcfd5b87ee60256 P 524dc62a7f9740b5b29c93bab94dafeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32392e70db0445fe9341ff89e046a25d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 37273 } } peers { permanent_uuid: "524dc62a7f9740b5b29c93bab94dafeb" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32923 } } peers { permanent_uuid: "1940c10288ad429e8bee620822bcd4e5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39535 } }
I20260928 06:33:50.614471 25059 raft_consensus.cc:374] T 31587501e8ab4fe88dcfd5b87ee60256 P 524dc62a7f9740b5b29c93bab94dafeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:50.614496 25059 raft_consensus.cc:729] T 31587501e8ab4fe88dcfd5b87ee60256 P 524dc62a7f9740b5b29c93bab94dafeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 524dc62a7f9740b5b29c93bab94dafeb, State: Initialized, Role: FOLLOWER
I20260928 06:33:50.614540 25059 consensus_queue.cc:260] T 31587501e8ab4fe88dcfd5b87ee60256 P 524dc62a7f9740b5b29c93bab94dafeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32392e70db0445fe9341ff89e046a25d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 37273 } } peers { permanent_uuid: "524dc62a7f9740b5b29c93bab94dafeb" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32923 } } peers { permanent_uuid: "1940c10288ad429e8bee620822bcd4e5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39535 } }
I20260928 06:33:50.614639 25059 ts_tablet_manager.cc:1428] T 31587501e8ab4fe88dcfd5b87ee60256 P 524dc62a7f9740b5b29c93bab94dafeb: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 06:33:50.614966 25082 ts_tablet_manager.cc:1918] T 06706edea8344ada8f0879fb044c9f2f P 1940c10288ad429e8bee620822bcd4e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:33:50.615077 25082 log.cc:1196] T 06706edea8344ada8f0879fb044c9f2f P 1940c10288ad429e8bee620822bcd4e5: Deleting WAL directory at /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/ts-2/wal/wals/06706edea8344ada8f0879fb044c9f2f
I20260928 06:33:50.615336 25082 ts_tablet_manager.cc:1939] T 06706edea8344ada8f0879fb044c9f2f P 1940c10288ad429e8bee620822bcd4e5: Deleting consensus metadata
I20260928 06:33:50.615816 24526 catalog_manager.cc:4841] TS 1940c10288ad429e8bee620822bcd4e5 (127.16.157.67:39535): tablet 06706edea8344ada8f0879fb044c9f2f (table test-table [id=a65ea94fe8fd4955bc3845b392b5bffb]) successfully deleted
I20260928 06:33:50.615947 25080 ts_tablet_manager.cc:1905] T 06706edea8344ada8f0879fb044c9f2f P 524dc62a7f9740b5b29c93bab94dafeb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:33:50.617410 25080 ts_tablet_manager.cc:1918] T 06706edea8344ada8f0879fb044c9f2f P 524dc62a7f9740b5b29c93bab94dafeb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:33:50.617477 25080 log.cc:1196] T 06706edea8344ada8f0879fb044c9f2f P 524dc62a7f9740b5b29c93bab94dafeb: Deleting WAL directory at /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/ts-1/wal/wals/06706edea8344ada8f0879fb044c9f2f
I20260928 06:33:50.617761 25080 ts_tablet_manager.cc:1939] T 06706edea8344ada8f0879fb044c9f2f P 524dc62a7f9740b5b29c93bab94dafeb: Deleting consensus metadata
I20260928 06:33:50.618155 24529 catalog_manager.cc:4841] TS 524dc62a7f9740b5b29c93bab94dafeb (127.16.157.66:32923): tablet 06706edea8344ada8f0879fb044c9f2f (table test-table [id=a65ea94fe8fd4955bc3845b392b5bffb]) successfully deleted
I20260928 06:33:50.621979 25058 tablet_bootstrap.cc:492] T 31587501e8ab4fe88dcfd5b87ee60256 P 32392e70db0445fe9341ff89e046a25d: No bootstrap required, opened a new log
I20260928 06:33:50.622036 25058 ts_tablet_manager.cc:1397] T 31587501e8ab4fe88dcfd5b87ee60256 P 32392e70db0445fe9341ff89e046a25d: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20260928 06:33:50.622165 25058 raft_consensus.cc:348] T 31587501e8ab4fe88dcfd5b87ee60256 P 32392e70db0445fe9341ff89e046a25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32392e70db0445fe9341ff89e046a25d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 37273 } } peers { permanent_uuid: "524dc62a7f9740b5b29c93bab94dafeb" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32923 } } peers { permanent_uuid: "1940c10288ad429e8bee620822bcd4e5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39535 } }
I20260928 06:33:50.622231 25058 raft_consensus.cc:374] T 31587501e8ab4fe88dcfd5b87ee60256 P 32392e70db0445fe9341ff89e046a25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:50.622251 25058 raft_consensus.cc:729] T 31587501e8ab4fe88dcfd5b87ee60256 P 32392e70db0445fe9341ff89e046a25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32392e70db0445fe9341ff89e046a25d, State: Initialized, Role: FOLLOWER
I20260928 06:33:50.622296 25058 consensus_queue.cc:260] T 31587501e8ab4fe88dcfd5b87ee60256 P 32392e70db0445fe9341ff89e046a25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32392e70db0445fe9341ff89e046a25d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 37273 } } peers { permanent_uuid: "524dc62a7f9740b5b29c93bab94dafeb" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32923 } } peers { permanent_uuid: "1940c10288ad429e8bee620822bcd4e5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39535 } }
I20260928 06:33:50.622519 25058 ts_tablet_manager.cc:1428] T 31587501e8ab4fe88dcfd5b87ee60256 P 32392e70db0445fe9341ff89e046a25d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
W20260928 06:33:50.643466 25030 tablet.cc:2369] T 31587501e8ab4fe88dcfd5b87ee60256 P 1940c10288ad429e8bee620822bcd4e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:33:50.673755 24888 tablet.cc:2369] T 31587501e8ab4fe88dcfd5b87ee60256 P 524dc62a7f9740b5b29c93bab94dafeb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:33:50.814034 25063 raft_consensus.cc:482] T 31587501e8ab4fe88dcfd5b87ee60256 P 32392e70db0445fe9341ff89e046a25d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:33:50.814147 25063 raft_consensus.cc:504] T 31587501e8ab4fe88dcfd5b87ee60256 P 32392e70db0445fe9341ff89e046a25d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32392e70db0445fe9341ff89e046a25d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 37273 } } peers { permanent_uuid: "524dc62a7f9740b5b29c93bab94dafeb" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32923 } } peers { permanent_uuid: "1940c10288ad429e8bee620822bcd4e5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39535 } }
I20260928 06:33:50.814566 25063 leader_election.cc:288] T 31587501e8ab4fe88dcfd5b87ee60256 P 32392e70db0445fe9341ff89e046a25d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 524dc62a7f9740b5b29c93bab94dafeb (127.16.157.66:32923), 1940c10288ad429e8bee620822bcd4e5 (127.16.157.67:39535)
I20260928 06:33:50.817747 25086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:50.814774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.66:32923 (local address 127.16.157.65:58751)
0928 06:33:50.814928 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:50.814944 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:50.815023 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:50.815222 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:50.815223 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:50.815225 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:33:50.815319 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:50.815321 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.816099 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.816100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:50.816908 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:50.816910 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.817462 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.817463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:50.817554 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:50.817558 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:33:50.817627 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 06:33:50.817668 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":84,"threads_started":1}
I20260928 06:33:50.817961 25055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:50.814969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:58751 (local address 127.16.157.66:32923)
0928 06:33:50.815019 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:50.815021 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:50.815034 (+ 13us) server_negotiation.cc:407] Connection header received
0928 06:33:50.815102 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:50.815103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:50.815116 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:50.815170 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:33:50.815402 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.816000 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.817005 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.817363 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.817443 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:50.817453 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:50.817856 (+ 403us) server_negotiation.cc:299] Negotiation successful
0928 06:33:50.817902 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 06:33:50.818423 25087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:50.814776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.67:39535 (local address 127.16.157.65:57017)
0928 06:33:50.815068 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:50.815079 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:50.815162 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:50.815311 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:50.815312 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:50.815313 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:33:50.815374 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:50.815376 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.816300 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.816301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:50.817394 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:50.817397 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.818159 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.818160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:50.818232 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:50.818237 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:33:50.818295 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 06:33:50.818341 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":3,"spinlock_wait_cycles":384,"thread_start_us":82,"threads_started":1}
I20260928 06:33:50.818427 25056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:50.814968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:57017 (local address 127.16.157.67:39535)
0928 06:33:50.815024 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:50.815026 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:50.815136 (+ 110us) server_negotiation.cc:407] Connection header received
0928 06:33:50.815226 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:50.815227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:50.815238 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:50.815296 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:33:50.815433 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.816211 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.817531 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.818077 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.818159 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:50.818166 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:50.818311 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 06:33:50.818355 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 06:33:50.818389 24840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31587501e8ab4fe88dcfd5b87ee60256" candidate_uuid: "32392e70db0445fe9341ff89e046a25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "524dc62a7f9740b5b29c93bab94dafeb" is_pre_election: true
I20260928 06:33:50.818553 24840 raft_consensus.cc:2454] T 31587501e8ab4fe88dcfd5b87ee60256 P 524dc62a7f9740b5b29c93bab94dafeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32392e70db0445fe9341ff89e046a25d in term 0.
I20260928 06:33:50.818707 24634 leader_election.cc:302] T 31587501e8ab4fe88dcfd5b87ee60256 P 32392e70db0445fe9341ff89e046a25d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32392e70db0445fe9341ff89e046a25d, 524dc62a7f9740b5b29c93bab94dafeb; no voters:
I20260928 06:33:50.818720 24982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31587501e8ab4fe88dcfd5b87ee60256" candidate_uuid: "32392e70db0445fe9341ff89e046a25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1940c10288ad429e8bee620822bcd4e5" is_pre_election: true
I20260928 06:33:50.818784 24982 raft_consensus.cc:2454] T 31587501e8ab4fe88dcfd5b87ee60256 P 1940c10288ad429e8bee620822bcd4e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32392e70db0445fe9341ff89e046a25d in term 0.
I20260928 06:33:50.818845 25063 raft_consensus.cc:2781] T 31587501e8ab4fe88dcfd5b87ee60256 P 32392e70db0445fe9341ff89e046a25d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:33:50.818881 25063 raft_consensus.cc:482] T 31587501e8ab4fe88dcfd5b87ee60256 P 32392e70db0445fe9341ff89e046a25d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:33:50.818902 25063 raft_consensus.cc:3037] T 31587501e8ab4fe88dcfd5b87ee60256 P 32392e70db0445fe9341ff89e046a25d [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:50.819599 25063 raft_consensus.cc:504] T 31587501e8ab4fe88dcfd5b87ee60256 P 32392e70db0445fe9341ff89e046a25d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32392e70db0445fe9341ff89e046a25d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 37273 } } peers { permanent_uuid: "524dc62a7f9740b5b29c93bab94dafeb" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32923 } } peers { permanent_uuid: "1940c10288ad429e8bee620822bcd4e5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39535 } }
I20260928 06:33:50.819729 25063 leader_election.cc:288] T 31587501e8ab4fe88dcfd5b87ee60256 P 32392e70db0445fe9341ff89e046a25d [CANDIDATE]: Term 1 election: Requested vote from peers 524dc62a7f9740b5b29c93bab94dafeb (127.16.157.66:32923), 1940c10288ad429e8bee620822bcd4e5 (127.16.157.67:39535)
I20260928 06:33:50.819911 24982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31587501e8ab4fe88dcfd5b87ee60256" candidate_uuid: "32392e70db0445fe9341ff89e046a25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1940c10288ad429e8bee620822bcd4e5"
I20260928 06:33:50.820010 24982 raft_consensus.cc:3037] T 31587501e8ab4fe88dcfd5b87ee60256 P 1940c10288ad429e8bee620822bcd4e5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:50.819998 24840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31587501e8ab4fe88dcfd5b87ee60256" candidate_uuid: "32392e70db0445fe9341ff89e046a25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "524dc62a7f9740b5b29c93bab94dafeb"
I20260928 06:33:50.820083 24840 raft_consensus.cc:3037] T 31587501e8ab4fe88dcfd5b87ee60256 P 524dc62a7f9740b5b29c93bab94dafeb [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:50.820672 24982 raft_consensus.cc:2454] T 31587501e8ab4fe88dcfd5b87ee60256 P 1940c10288ad429e8bee620822bcd4e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32392e70db0445fe9341ff89e046a25d in term 1.
I20260928 06:33:50.820674 24840 raft_consensus.cc:2454] T 31587501e8ab4fe88dcfd5b87ee60256 P 524dc62a7f9740b5b29c93bab94dafeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32392e70db0445fe9341ff89e046a25d in term 1.
I20260928 06:33:50.820899 24631 leader_election.cc:302] T 31587501e8ab4fe88dcfd5b87ee60256 P 32392e70db0445fe9341ff89e046a25d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1940c10288ad429e8bee620822bcd4e5, 32392e70db0445fe9341ff89e046a25d; no voters:
I20260928 06:33:50.821035 25063 raft_consensus.cc:2781] T 31587501e8ab4fe88dcfd5b87ee60256 P 32392e70db0445fe9341ff89e046a25d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:33:50.821179 25063 raft_consensus.cc:686] T 31587501e8ab4fe88dcfd5b87ee60256 P 32392e70db0445fe9341ff89e046a25d [term 1 LEADER]: Becoming Leader. State: Replica: 32392e70db0445fe9341ff89e046a25d, State: Running, Role: LEADER
I20260928 06:33:50.821255 25063 consensus_queue.cc:237] T 31587501e8ab4fe88dcfd5b87ee60256 P 32392e70db0445fe9341ff89e046a25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32392e70db0445fe9341ff89e046a25d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 37273 } } peers { permanent_uuid: "524dc62a7f9740b5b29c93bab94dafeb" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32923 } } peers { permanent_uuid: "1940c10288ad429e8bee620822bcd4e5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39535 } }
I20260928 06:33:50.822002 24542 catalog_manager.cc:5495] T 31587501e8ab4fe88dcfd5b87ee60256 P 32392e70db0445fe9341ff89e046a25d reported cstate change: term changed from 0 to 1, leader changed from <none> to 32392e70db0445fe9341ff89e046a25d (127.16.157.65). New cstate: current_term: 1 leader_uuid: "32392e70db0445fe9341ff89e046a25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32392e70db0445fe9341ff89e046a25d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 37273 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "524dc62a7f9740b5b29c93bab94dafeb" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1940c10288ad429e8bee620822bcd4e5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39535 } health_report { overall_health: UNKNOWN } } }
Sep 28 06:33:50 dist-test-slave-rrj5 krb5kdc[24497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577229, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.157.126@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM
I20260928 06:33:50.862761 24593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:50.854304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51550 (local address 127.16.157.126:44663)
0928 06:33:50.854360 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:50.854362 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:50.854513 (+ 151us) server_negotiation.cc:407] Connection header received
0928 06:33:50.854587 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:50.854588 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:50.854598 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:50.854692 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:50.855305 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.856069 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.857071 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.857339 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.857402 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:50.859964 (+ 2562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:50.859965 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:50.859966 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:50.859972 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:50.861635 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:50.862073 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:50.862074 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:50.862074 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:50.862125 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:50.862325 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:50.862326 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:50.862326 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:50.862420 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:50.862473 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:50.862635 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 06:33:50.862698 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 06:33:50.862869 25095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:50.854246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:44663 (local address 127.0.0.1:51550)
0928 06:33:50.854404 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:50.854424 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:50.854530 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:50.854701 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:50.854702 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:50.855145 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:50.855206 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:50.855209 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.856161 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.856162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:50.856993 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:50.856995 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.857419 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.857420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:50.857472 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:50.858175 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:50.858180 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:50.859878 (+ 1698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:50.861716 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:50.861725 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:50.861727 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:50.861993 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:50.862188 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:50.862189 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:50.862189 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:50.862262 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:50.862482 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:50.862489 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:50.862540 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 06:33:50.862618 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 06:33:50.862768 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":38,"thread_start_us":81,"threads_started":1}
I20260928 06:33:50.864979 24542 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.16.157.126@KRBTEST.COM'} at 127.0.0.1:51550:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 06:33:50.865207 24542 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:33:50.866590 24812 tablet_service.cc:1467] Processing CreateTablet for tablet 30c7e5f77d12430aa29eaad3339b697c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d5fcde1ff40948a3b16e5a84f190a80c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:33:50.866600 24962 tablet_service.cc:1467] Processing CreateTablet for tablet 30c7e5f77d12430aa29eaad3339b697c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d5fcde1ff40948a3b16e5a84f190a80c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:33:50.866636 24680 tablet_service.cc:1467] Processing CreateTablet for tablet 30c7e5f77d12430aa29eaad3339b697c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d5fcde1ff40948a3b16e5a84f190a80c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:33:50.866743 24962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30c7e5f77d12430aa29eaad3339b697c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:50.866743 24812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30c7e5f77d12430aa29eaad3339b697c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:50.866791 24680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30c7e5f77d12430aa29eaad3339b697c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:50.868217 25062 tablet_bootstrap.cc:492] T 30c7e5f77d12430aa29eaad3339b697c P 1940c10288ad429e8bee620822bcd4e5: Bootstrap starting.
I20260928 06:33:50.868274 25059 tablet_bootstrap.cc:492] T 30c7e5f77d12430aa29eaad3339b697c P 524dc62a7f9740b5b29c93bab94dafeb: Bootstrap starting.
I20260928 06:33:50.868589 25062 tablet_bootstrap.cc:654] T 30c7e5f77d12430aa29eaad3339b697c P 1940c10288ad429e8bee620822bcd4e5: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:50.868597 25059 tablet_bootstrap.cc:654] T 30c7e5f77d12430aa29eaad3339b697c P 524dc62a7f9740b5b29c93bab94dafeb: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:50.868846 25058 tablet_bootstrap.cc:492] T 30c7e5f77d12430aa29eaad3339b697c P 32392e70db0445fe9341ff89e046a25d: Bootstrap starting.
I20260928 06:33:50.869230 25058 tablet_bootstrap.cc:654] T 30c7e5f77d12430aa29eaad3339b697c P 32392e70db0445fe9341ff89e046a25d: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:50.869262 25062 tablet_bootstrap.cc:492] T 30c7e5f77d12430aa29eaad3339b697c P 1940c10288ad429e8bee620822bcd4e5: No bootstrap required, opened a new log
I20260928 06:33:50.869328 25062 ts_tablet_manager.cc:1397] T 30c7e5f77d12430aa29eaad3339b697c P 1940c10288ad429e8bee620822bcd4e5: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:50.869482 25062 raft_consensus.cc:348] T 30c7e5f77d12430aa29eaad3339b697c P 1940c10288ad429e8bee620822bcd4e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32392e70db0445fe9341ff89e046a25d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 37273 } } peers { permanent_uuid: "524dc62a7f9740b5b29c93bab94dafeb" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32923 } } peers { permanent_uuid: "1940c10288ad429e8bee620822bcd4e5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39535 } }
I20260928 06:33:50.869596 25062 raft_consensus.cc:374] T 30c7e5f77d12430aa29eaad3339b697c P 1940c10288ad429e8bee620822bcd4e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:50.869626 25062 raft_consensus.cc:729] T 30c7e5f77d12430aa29eaad3339b697c P 1940c10288ad429e8bee620822bcd4e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1940c10288ad429e8bee620822bcd4e5, State: Initialized, Role: FOLLOWER
I20260928 06:33:50.869668 25062 consensus_queue.cc:260] T 30c7e5f77d12430aa29eaad3339b697c P 1940c10288ad429e8bee620822bcd4e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32392e70db0445fe9341ff89e046a25d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 37273 } } peers { permanent_uuid: "524dc62a7f9740b5b29c93bab94dafeb" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32923 } } peers { permanent_uuid: "1940c10288ad429e8bee620822bcd4e5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39535 } }
I20260928 06:33:50.869793 25062 ts_tablet_manager.cc:1428] T 30c7e5f77d12430aa29eaad3339b697c P 1940c10288ad429e8bee620822bcd4e5: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 06:33:50.869886 25059 tablet_bootstrap.cc:492] T 30c7e5f77d12430aa29eaad3339b697c P 524dc62a7f9740b5b29c93bab94dafeb: No bootstrap required, opened a new log
I20260928 06:33:50.869954 25059 ts_tablet_manager.cc:1397] T 30c7e5f77d12430aa29eaad3339b697c P 524dc62a7f9740b5b29c93bab94dafeb: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 06:33:50.869892 25066 tablet_replica.cc:439] T 30c7e5f77d12430aa29eaad3339b697c P 1940c10288ad429e8bee620822bcd4e5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32392e70db0445fe9341ff89e046a25d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 37273 } } peers { permanent_uuid: "524dc62a7f9740b5b29c93bab94dafeb" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32923 } } peers { permanent_uuid: "1940c10288ad429e8bee620822bcd4e5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39535 } } }
I20260928 06:33:50.869993 25066 tablet_replica.cc:442] T 30c7e5f77d12430aa29eaad3339b697c P 1940c10288ad429e8bee620822bcd4e5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:33:50.870066 25059 raft_consensus.cc:348] T 30c7e5f77d12430aa29eaad3339b697c P 524dc62a7f9740b5b29c93bab94dafeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32392e70db0445fe9341ff89e046a25d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 37273 } } peers { permanent_uuid: "524dc62a7f9740b5b29c93bab94dafeb" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32923 } } peers { permanent_uuid: "1940c10288ad429e8bee620822bcd4e5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39535 } }
I20260928 06:33:50.870146 25059 raft_consensus.cc:374] T 30c7e5f77d12430aa29eaad3339b697c P 524dc62a7f9740b5b29c93bab94dafeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:50.870199 25059 raft_consensus.cc:729] T 30c7e5f77d12430aa29eaad3339b697c P 524dc62a7f9740b5b29c93bab94dafeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 524dc62a7f9740b5b29c93bab94dafeb, State: Initialized, Role: FOLLOWER
I20260928 06:33:50.870267 25059 consensus_queue.cc:260] T 30c7e5f77d12430aa29eaad3339b697c P 524dc62a7f9740b5b29c93bab94dafeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32392e70db0445fe9341ff89e046a25d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 37273 } } peers { permanent_uuid: "524dc62a7f9740b5b29c93bab94dafeb" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32923 } } peers { permanent_uuid: "1940c10288ad429e8bee620822bcd4e5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39535 } }
I20260928 06:33:50.870390 25059 ts_tablet_manager.cc:1428] T 30c7e5f77d12430aa29eaad3339b697c P 524dc62a7f9740b5b29c93bab94dafeb: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 06:33:50.870594 25065 tablet_replica.cc:439] T 30c7e5f77d12430aa29eaad3339b697c P 524dc62a7f9740b5b29c93bab94dafeb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32392e70db0445fe9341ff89e046a25d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 37273 } } peers { permanent_uuid: "524dc62a7f9740b5b29c93bab94dafeb" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32923 } } peers { permanent_uuid: "1940c10288ad429e8bee620822bcd4e5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39535 } } }
I20260928 06:33:50.870688 25065 tablet_replica.cc:442] T 30c7e5f77d12430aa29eaad3339b697c P 524dc62a7f9740b5b29c93bab94dafeb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:33:50.871011 25058 tablet_bootstrap.cc:492] T 30c7e5f77d12430aa29eaad3339b697c P 32392e70db0445fe9341ff89e046a25d: No bootstrap required, opened a new log
I20260928 06:33:50.871066 25058 ts_tablet_manager.cc:1397] T 30c7e5f77d12430aa29eaad3339b697c P 32392e70db0445fe9341ff89e046a25d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:33:50.871222 25058 raft_consensus.cc:348] T 30c7e5f77d12430aa29eaad3339b697c P 32392e70db0445fe9341ff89e046a25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32392e70db0445fe9341ff89e046a25d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 37273 } } peers { permanent_uuid: "524dc62a7f9740b5b29c93bab94dafeb" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32923 } } peers { permanent_uuid: "1940c10288ad429e8bee620822bcd4e5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39535 } }
I20260928 06:33:50.871304 25058 raft_consensus.cc:374] T 30c7e5f77d12430aa29eaad3339b697c P 32392e70db0445fe9341ff89e046a25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:50.871327 25058 raft_consensus.cc:729] T 30c7e5f77d12430aa29eaad3339b697c P 32392e70db0445fe9341ff89e046a25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32392e70db0445fe9341ff89e046a25d, State: Initialized, Role: FOLLOWER
I20260928 06:33:50.871374 25058 consensus_queue.cc:260] T 30c7e5f77d12430aa29eaad3339b697c P 32392e70db0445fe9341ff89e046a25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32392e70db0445fe9341ff89e046a25d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 37273 } } peers { permanent_uuid: "524dc62a7f9740b5b29c93bab94dafeb" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32923 } } peers { permanent_uuid: "1940c10288ad429e8bee620822bcd4e5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39535 } }
I20260928 06:33:50.871472 25058 ts_tablet_manager.cc:1428] T 30c7e5f77d12430aa29eaad3339b697c P 32392e70db0445fe9341ff89e046a25d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:50.871630 25063 tablet_replica.cc:439] T 30c7e5f77d12430aa29eaad3339b697c P 32392e70db0445fe9341ff89e046a25d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32392e70db0445fe9341ff89e046a25d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 37273 } } peers { permanent_uuid: "524dc62a7f9740b5b29c93bab94dafeb" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32923 } } peers { permanent_uuid: "1940c10288ad429e8bee620822bcd4e5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39535 } } }
I20260928 06:33:50.871718 25063 tablet_replica.cc:442] T 30c7e5f77d12430aa29eaad3339b697c P 32392e70db0445fe9341ff89e046a25d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:33:50.942469 25065 raft_consensus.cc:482] T 30c7e5f77d12430aa29eaad3339b697c P 524dc62a7f9740b5b29c93bab94dafeb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:33:50.942559 25065 raft_consensus.cc:504] T 30c7e5f77d12430aa29eaad3339b697c P 524dc62a7f9740b5b29c93bab94dafeb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32392e70db0445fe9341ff89e046a25d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 37273 } } peers { permanent_uuid: "524dc62a7f9740b5b29c93bab94dafeb" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32923 } } peers { permanent_uuid: "1940c10288ad429e8bee620822bcd4e5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39535 } }
I20260928 06:33:50.942731 25065 leader_election.cc:288] T 30c7e5f77d12430aa29eaad3339b697c P 524dc62a7f9740b5b29c93bab94dafeb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 32392e70db0445fe9341ff89e046a25d (127.16.157.65:37273), 1940c10288ad429e8bee620822bcd4e5 (127.16.157.67:39535)
I20260928 06:33:50.942948 24701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c7e5f77d12430aa29eaad3339b697c" candidate_uuid: "524dc62a7f9740b5b29c93bab94dafeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32392e70db0445fe9341ff89e046a25d" is_pre_election: true
I20260928 06:33:50.943048 24701 raft_consensus.cc:2454] T 30c7e5f77d12430aa29eaad3339b697c P 32392e70db0445fe9341ff89e046a25d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 524dc62a7f9740b5b29c93bab94dafeb in term 0.
I20260928 06:33:50.942998 24982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c7e5f77d12430aa29eaad3339b697c" candidate_uuid: "524dc62a7f9740b5b29c93bab94dafeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1940c10288ad429e8bee620822bcd4e5" is_pre_election: true
I20260928 06:33:50.943094 24982 raft_consensus.cc:2454] T 30c7e5f77d12430aa29eaad3339b697c P 1940c10288ad429e8bee620822bcd4e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 524dc62a7f9740b5b29c93bab94dafeb in term 0.
I20260928 06:33:50.943205 24774 leader_election.cc:302] T 30c7e5f77d12430aa29eaad3339b697c P 524dc62a7f9740b5b29c93bab94dafeb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32392e70db0445fe9341ff89e046a25d, 524dc62a7f9740b5b29c93bab94dafeb; no voters:
I20260928 06:33:50.943289 25065 raft_consensus.cc:2781] T 30c7e5f77d12430aa29eaad3339b697c P 524dc62a7f9740b5b29c93bab94dafeb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:33:50.943320 25065 raft_consensus.cc:482] T 30c7e5f77d12430aa29eaad3339b697c P 524dc62a7f9740b5b29c93bab94dafeb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:33:50.943377 25065 raft_consensus.cc:3037] T 30c7e5f77d12430aa29eaad3339b697c P 524dc62a7f9740b5b29c93bab94dafeb [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:50.944010 25065 raft_consensus.cc:504] T 30c7e5f77d12430aa29eaad3339b697c P 524dc62a7f9740b5b29c93bab94dafeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32392e70db0445fe9341ff89e046a25d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 37273 } } peers { permanent_uuid: "524dc62a7f9740b5b29c93bab94dafeb" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32923 } } peers { permanent_uuid: "1940c10288ad429e8bee620822bcd4e5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39535 } }
I20260928 06:33:50.944146 25065 leader_election.cc:288] T 30c7e5f77d12430aa29eaad3339b697c P 524dc62a7f9740b5b29c93bab94dafeb [CANDIDATE]: Term 1 election: Requested vote from peers 32392e70db0445fe9341ff89e046a25d (127.16.157.65:37273), 1940c10288ad429e8bee620822bcd4e5 (127.16.157.67:39535)
I20260928 06:33:50.944290 24701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c7e5f77d12430aa29eaad3339b697c" candidate_uuid: "524dc62a7f9740b5b29c93bab94dafeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32392e70db0445fe9341ff89e046a25d"
I20260928 06:33:50.944383 24701 raft_consensus.cc:3037] T 30c7e5f77d12430aa29eaad3339b697c P 32392e70db0445fe9341ff89e046a25d [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:50.944375 24982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c7e5f77d12430aa29eaad3339b697c" candidate_uuid: "524dc62a7f9740b5b29c93bab94dafeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1940c10288ad429e8bee620822bcd4e5"
I20260928 06:33:50.944439 24982 raft_consensus.cc:3037] T 30c7e5f77d12430aa29eaad3339b697c P 1940c10288ad429e8bee620822bcd4e5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:50.944988 24982 raft_consensus.cc:2454] T 30c7e5f77d12430aa29eaad3339b697c P 1940c10288ad429e8bee620822bcd4e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 524dc62a7f9740b5b29c93bab94dafeb in term 1.
I20260928 06:33:50.945127 24701 raft_consensus.cc:2454] T 30c7e5f77d12430aa29eaad3339b697c P 32392e70db0445fe9341ff89e046a25d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 524dc62a7f9740b5b29c93bab94dafeb in term 1.
I20260928 06:33:50.945166 24772 leader_election.cc:302] T 30c7e5f77d12430aa29eaad3339b697c P 524dc62a7f9740b5b29c93bab94dafeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1940c10288ad429e8bee620822bcd4e5, 524dc62a7f9740b5b29c93bab94dafeb; no voters:
I20260928 06:33:50.945246 25065 raft_consensus.cc:2781] T 30c7e5f77d12430aa29eaad3339b697c P 524dc62a7f9740b5b29c93bab94dafeb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:33:50.945287 25065 raft_consensus.cc:686] T 30c7e5f77d12430aa29eaad3339b697c P 524dc62a7f9740b5b29c93bab94dafeb [term 1 LEADER]: Becoming Leader. State: Replica: 524dc62a7f9740b5b29c93bab94dafeb, State: Running, Role: LEADER
I20260928 06:33:50.945348 25065 consensus_queue.cc:237] T 30c7e5f77d12430aa29eaad3339b697c P 524dc62a7f9740b5b29c93bab94dafeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32392e70db0445fe9341ff89e046a25d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 37273 } } peers { permanent_uuid: "524dc62a7f9740b5b29c93bab94dafeb" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32923 } } peers { permanent_uuid: "1940c10288ad429e8bee620822bcd4e5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39535 } }
I20260928 06:33:50.945652 25068 tablet_replica.cc:439] T 30c7e5f77d12430aa29eaad3339b697c P 524dc62a7f9740b5b29c93bab94dafeb: TxnStatusTablet state changed. Reason: New leader 524dc62a7f9740b5b29c93bab94dafeb. Latest consensus state: current_term: 1 leader_uuid: "524dc62a7f9740b5b29c93bab94dafeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32392e70db0445fe9341ff89e046a25d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 37273 } } peers { permanent_uuid: "524dc62a7f9740b5b29c93bab94dafeb" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32923 } } peers { permanent_uuid: "1940c10288ad429e8bee620822bcd4e5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39535 } } }
I20260928 06:33:50.945737 25068 tablet_replica.cc:442] T 30c7e5f77d12430aa29eaad3339b697c P 524dc62a7f9740b5b29c93bab94dafeb: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:33:50.945890 25100 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 06:33:50.946012 25100 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 06:33:50.945940 24542 catalog_manager.cc:5495] T 30c7e5f77d12430aa29eaad3339b697c P 524dc62a7f9740b5b29c93bab94dafeb reported cstate change: term changed from 0 to 1, leader changed from <none> to 524dc62a7f9740b5b29c93bab94dafeb (127.16.157.66). New cstate: current_term: 1 leader_uuid: "524dc62a7f9740b5b29c93bab94dafeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32392e70db0445fe9341ff89e046a25d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 37273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "524dc62a7f9740b5b29c93bab94dafeb" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1940c10288ad429e8bee620822bcd4e5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39535 } health_report { overall_health: UNKNOWN } } }
I20260928 06:33:50.967170 25055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:50.963986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43012 (local address 127.16.157.66:32923)
0928 06:33:50.964034 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:50.964036 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:50.964155 (+ 119us) server_negotiation.cc:407] Connection header received
0928 06:33:50.964225 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:50.964226 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:50.964237 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:50.964299 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:33:50.964426 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.965167 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.966307 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.966616 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.966688 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:50.966903 (+ 215us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:33:50.966940 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:33:50.966995 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:50.967070 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 06:33:50.967092 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 06:33:50.967217 25095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:50.963935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.66:32923 (local address 127.0.0.1:43012)
0928 06:33:50.964090 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:50.964102 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:50.964159 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:50.964305 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:50.964306 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:50.964308 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:33:50.964362 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:50.964365 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.965252 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.965253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:50.966210 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:50.966213 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.966710 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.966712 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:50.966782 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:50.966801 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:33:50.967007 (+ 206us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:33:50.967016 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 06:33:50.967066 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 06:33:50.967123 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116}
I20260928 06:33:50.968207 24701 raft_consensus.cc:1261] T 30c7e5f77d12430aa29eaad3339b697c P 32392e70db0445fe9341ff89e046a25d [term 1 FOLLOWER]: Refusing update from remote peer 524dc62a7f9740b5b29c93bab94dafeb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:33:50.968206 24982 raft_consensus.cc:1261] T 30c7e5f77d12430aa29eaad3339b697c P 1940c10288ad429e8bee620822bcd4e5 [term 1 FOLLOWER]: Refusing update from remote peer 524dc62a7f9740b5b29c93bab94dafeb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:33:50.968415 25068 consensus_queue.cc:1035] T 30c7e5f77d12430aa29eaad3339b697c P 524dc62a7f9740b5b29c93bab94dafeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "1940c10288ad429e8bee620822bcd4e5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:33:50.968524 25065 consensus_queue.cc:1035] T 30c7e5f77d12430aa29eaad3339b697c P 524dc62a7f9740b5b29c93bab94dafeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "32392e70db0445fe9341ff89e046a25d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 37273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:33:50.968865 25066 tablet_replica.cc:439] T 30c7e5f77d12430aa29eaad3339b697c P 1940c10288ad429e8bee620822bcd4e5: TxnStatusTablet state changed. Reason: New leader 524dc62a7f9740b5b29c93bab94dafeb. Latest consensus state: current_term: 1 leader_uuid: "524dc62a7f9740b5b29c93bab94dafeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32392e70db0445fe9341ff89e046a25d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 37273 } } peers { permanent_uuid: "524dc62a7f9740b5b29c93bab94dafeb" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32923 } } peers { permanent_uuid: "1940c10288ad429e8bee620822bcd4e5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39535 } } }
I20260928 06:33:50.968961 25066 tablet_replica.cc:442] T 30c7e5f77d12430aa29eaad3339b697c P 1940c10288ad429e8bee620822bcd4e5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:33:50.969000 25063 tablet_replica.cc:439] T 30c7e5f77d12430aa29eaad3339b697c P 32392e70db0445fe9341ff89e046a25d: TxnStatusTablet state changed. Reason: New leader 524dc62a7f9740b5b29c93bab94dafeb. Latest consensus state: current_term: 1 leader_uuid: "524dc62a7f9740b5b29c93bab94dafeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32392e70db0445fe9341ff89e046a25d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 37273 } } peers { permanent_uuid: "524dc62a7f9740b5b29c93bab94dafeb" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32923 } } peers { permanent_uuid: "1940c10288ad429e8bee620822bcd4e5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39535 } } }
I20260928 06:33:50.969076 25063 tablet_replica.cc:442] T 30c7e5f77d12430aa29eaad3339b697c P 32392e70db0445fe9341ff89e046a25d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:33:50.969605 25065 tablet_replica.cc:439] T 30c7e5f77d12430aa29eaad3339b697c P 524dc62a7f9740b5b29c93bab94dafeb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "524dc62a7f9740b5b29c93bab94dafeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32392e70db0445fe9341ff89e046a25d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 37273 } } peers { permanent_uuid: "524dc62a7f9740b5b29c93bab94dafeb" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32923 } } peers { permanent_uuid: "1940c10288ad429e8bee620822bcd4e5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39535 } } }
I20260928 06:33:50.969681 25065 tablet_replica.cc:442] T 30c7e5f77d12430aa29eaad3339b697c P 524dc62a7f9740b5b29c93bab94dafeb: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:33:50.970316 25063 tablet_replica.cc:439] T 30c7e5f77d12430aa29eaad3339b697c P 32392e70db0445fe9341ff89e046a25d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "524dc62a7f9740b5b29c93bab94dafeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32392e70db0445fe9341ff89e046a25d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 37273 } } peers { permanent_uuid: "524dc62a7f9740b5b29c93bab94dafeb" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32923 } } peers { permanent_uuid: "1940c10288ad429e8bee620822bcd4e5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39535 } } }
I20260928 06:33:50.970397 25063 tablet_replica.cc:442] T 30c7e5f77d12430aa29eaad3339b697c P 32392e70db0445fe9341ff89e046a25d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:33:50.970746 25068 tablet_replica.cc:439] T 30c7e5f77d12430aa29eaad3339b697c P 524dc62a7f9740b5b29c93bab94dafeb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "524dc62a7f9740b5b29c93bab94dafeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32392e70db0445fe9341ff89e046a25d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 37273 } } peers { permanent_uuid: "524dc62a7f9740b5b29c93bab94dafeb" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32923 } } peers { permanent_uuid: "1940c10288ad429e8bee620822bcd4e5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39535 } } }
I20260928 06:33:50.970819 25068 tablet_replica.cc:442] T 30c7e5f77d12430aa29eaad3339b697c P 524dc62a7f9740b5b29c93bab94dafeb: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:33:50.971221 25073 mvcc.cc:205] Tried to move back new op lower bound from 7334204338044133376 to 7334204337952448512. Current Snapshot: MvccSnapshot[applied={T|T < 7334204338044133376}]
I20260928 06:33:50.971393 25066 tablet_replica.cc:439] T 30c7e5f77d12430aa29eaad3339b697c P 1940c10288ad429e8bee620822bcd4e5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "524dc62a7f9740b5b29c93bab94dafeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32392e70db0445fe9341ff89e046a25d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 37273 } } peers { permanent_uuid: "524dc62a7f9740b5b29c93bab94dafeb" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32923 } } peers { permanent_uuid: "1940c10288ad429e8bee620822bcd4e5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39535 } } }
I20260928 06:33:50.971464 25066 tablet_replica.cc:442] T 30c7e5f77d12430aa29eaad3339b697c P 1940c10288ad429e8bee620822bcd4e5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:33:50.976197 25054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:50.973071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34292 (local address 127.16.157.65:37273)
0928 06:33:50.973114 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:50.973116 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:50.973124 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:33:50.973190 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:50.973191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:50.973210 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:50.973283 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:33:50.973401 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.974145 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.975228 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.975538 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.975603 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:50.975816 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:33:50.975947 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:33:50.976020 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:50.976089 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 06:33:50.976110 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:33:50.982116 25055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:50.979129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:51221 (local address 127.16.157.66:32923)
0928 06:33:50.979164 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:50.979166 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:50.979174 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:33:50.979237 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:50.979237 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:50.979247 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:50.979304 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:33:50.979456 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.980146 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.981283 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.981610 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.981680 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:50.981870 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:33:50.981903 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:33:50.981956 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:50.982029 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 06:33:50.982049 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 06:33:50.982154 25107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:50.978956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.66:32923 (local address 127.16.157.65:51221)
0928 06:33:50.979116 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:50.979128 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:50.979182 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:50.979318 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:50.979319 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:50.979320 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:33:50.979385 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:50.979386 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.980244 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.980245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:50.981191 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:50.981193 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.981691 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.981692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:50.981765 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:50.981781 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:33:50.981973 (+ 192us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:33:50.981975 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:33:50.982017 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 06:33:50.982063 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":78,"threads_started":1}
I20260928 06:33:50.984671 24982 raft_consensus.cc:1261] T 31587501e8ab4fe88dcfd5b87ee60256 P 1940c10288ad429e8bee620822bcd4e5 [term 1 FOLLOWER]: Refusing update from remote peer 32392e70db0445fe9341ff89e046a25d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:33:50.984706 24840 raft_consensus.cc:1261] T 31587501e8ab4fe88dcfd5b87ee60256 P 524dc62a7f9740b5b29c93bab94dafeb [term 1 FOLLOWER]: Refusing update from remote peer 32392e70db0445fe9341ff89e046a25d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:33:50.985008 25088 consensus_queue.cc:1035] T 31587501e8ab4fe88dcfd5b87ee60256 P 32392e70db0445fe9341ff89e046a25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1940c10288ad429e8bee620822bcd4e5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:33:50.985132 25063 consensus_queue.cc:1035] T 31587501e8ab4fe88dcfd5b87ee60256 P 32392e70db0445fe9341ff89e046a25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "524dc62a7f9740b5b29c93bab94dafeb" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:33:50.994056 25111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:50.990865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.65:37273 (local address 127.16.157.66:43225)
0928 06:33:50.991026 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:50.991038 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:50.991104 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:50.991263 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:50.991264 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:50.991265 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:33:50.991311 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:50.991312 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.992218 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.992219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:50.993129 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:50.993132 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.993632 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.993633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:50.993687 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:50.993701 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:33:50.993880 (+ 179us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:33:50.993881 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 06:33:50.993934 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 06:33:50.993976 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":102,"threads_started":1}
I20260928 06:33:50.994510 25054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:50.990918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.66:43225 (local address 127.16.157.65:37273)
0928 06:33:50.990951 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:50.990953 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:50.991104 (+ 151us) server_negotiation.cc:407] Connection header received
0928 06:33:50.991164 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:50.991165 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:50.991175 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:50.991224 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:33:50.991383 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.992116 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.993226 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:50.993516 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:50.993554 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:50.993786 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:33:50.993817 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:33:50.993843 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:50.994425 (+ 582us) server_negotiation.cc:299] Negotiation successful
0928 06:33:50.994447 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 06:33:51.002956 24540 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:33:51.003077 24540 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:33:51.004582 24540 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 217c529c34ff472987667c1af6f76a6b: Sending DeleteTablet for 3 replicas of tablet 31587501e8ab4fe88dcfd5b87ee60256
I20260928 06:33:51.004920 24812 tablet_service.cc:1514] Processing DeleteTablet for tablet 31587501e8ab4fe88dcfd5b87ee60256 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:33:51 UTC) from {username='oryx', principal='oryx/127.16.157.126@KRBTEST.COM'} at 127.0.0.1:42992
I20260928 06:33:51.005023 25080 tablet_replica.cc:330] T 31587501e8ab4fe88dcfd5b87ee60256 P 524dc62a7f9740b5b29c93bab94dafeb: stopping tablet replica
I20260928 06:33:51.005091 25080 raft_consensus.cc:2229] T 31587501e8ab4fe88dcfd5b87ee60256 P 524dc62a7f9740b5b29c93bab94dafeb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:33:51.005129 25080 raft_consensus.cc:2258] T 31587501e8ab4fe88dcfd5b87ee60256 P 524dc62a7f9740b5b29c93bab94dafeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:33:51.005115 24680 tablet_service.cc:1514] Processing DeleteTablet for tablet 31587501e8ab4fe88dcfd5b87ee60256 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:33:51 UTC) from {username='oryx', principal='oryx/127.16.157.126@KRBTEST.COM'} at 127.0.0.1:34288
I20260928 06:33:51.005117 24962 tablet_service.cc:1514] Processing DeleteTablet for tablet 31587501e8ab4fe88dcfd5b87ee60256 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:33:51 UTC) from {username='oryx', principal='oryx/127.16.157.126@KRBTEST.COM'} at 127.0.0.1:38864
I20260928 06:33:51.005260 25082 tablet_replica.cc:330] T 31587501e8ab4fe88dcfd5b87ee60256 P 1940c10288ad429e8bee620822bcd4e5: stopping tablet replica
I20260928 06:33:51.005270 25081 tablet_replica.cc:330] T 31587501e8ab4fe88dcfd5b87ee60256 P 32392e70db0445fe9341ff89e046a25d: stopping tablet replica
I20260928 06:33:51.005306 25081 raft_consensus.cc:2229] T 31587501e8ab4fe88dcfd5b87ee60256 P 32392e70db0445fe9341ff89e046a25d [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:33:51.005314 25082 raft_consensus.cc:2229] T 31587501e8ab4fe88dcfd5b87ee60256 P 1940c10288ad429e8bee620822bcd4e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:33:51.005348 25082 raft_consensus.cc:2258] T 31587501e8ab4fe88dcfd5b87ee60256 P 1940c10288ad429e8bee620822bcd4e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:33:51.005407 25081 raft_consensus.cc:2258] T 31587501e8ab4fe88dcfd5b87ee60256 P 32392e70db0445fe9341ff89e046a25d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:33:51.005683 25082 ts_tablet_manager.cc:1905] T 31587501e8ab4fe88dcfd5b87ee60256 P 1940c10288ad429e8bee620822bcd4e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:33:51.005947 25081 ts_tablet_manager.cc:1905] T 31587501e8ab4fe88dcfd5b87ee60256 P 32392e70db0445fe9341ff89e046a25d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:33:51.007308 25082 ts_tablet_manager.cc:1918] T 31587501e8ab4fe88dcfd5b87ee60256 P 1940c10288ad429e8bee620822bcd4e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:33:51.007359 25082 log.cc:1196] T 31587501e8ab4fe88dcfd5b87ee60256 P 1940c10288ad429e8bee620822bcd4e5: Deleting WAL directory at /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/ts-2/wal/wals/31587501e8ab4fe88dcfd5b87ee60256
I20260928 06:33:51.007490 25081 ts_tablet_manager.cc:1918] T 31587501e8ab4fe88dcfd5b87ee60256 P 32392e70db0445fe9341ff89e046a25d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:33:51.007537 25081 log.cc:1196] T 31587501e8ab4fe88dcfd5b87ee60256 P 32392e70db0445fe9341ff89e046a25d: Deleting WAL directory at /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/ts-0/wal/wals/31587501e8ab4fe88dcfd5b87ee60256
I20260928 06:33:51.007582 25082 ts_tablet_manager.cc:1939] T 31587501e8ab4fe88dcfd5b87ee60256 P 1940c10288ad429e8bee620822bcd4e5: Deleting consensus metadata
I20260928 06:33:51.007738 25081 ts_tablet_manager.cc:1939] T 31587501e8ab4fe88dcfd5b87ee60256 P 32392e70db0445fe9341ff89e046a25d: Deleting consensus metadata
I20260928 06:33:51.007972 24526 catalog_manager.cc:4841] TS 1940c10288ad429e8bee620822bcd4e5 (127.16.157.67:39535): tablet 31587501e8ab4fe88dcfd5b87ee60256 (table test-table [id=ff733fa3c82f4ddfa814e02056ba27f1]) successfully deleted
I20260928 06:33:51.008010 24528 catalog_manager.cc:4841] TS 32392e70db0445fe9341ff89e046a25d (127.16.157.65:37273): tablet 31587501e8ab4fe88dcfd5b87ee60256 (table test-table [id=ff733fa3c82f4ddfa814e02056ba27f1]) successfully deleted
I20260928 06:33:51.008208 25080 ts_tablet_manager.cc:1905] T 31587501e8ab4fe88dcfd5b87ee60256 P 524dc62a7f9740b5b29c93bab94dafeb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:33:51.009656 25080 ts_tablet_manager.cc:1918] T 31587501e8ab4fe88dcfd5b87ee60256 P 524dc62a7f9740b5b29c93bab94dafeb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:33:51.009717 25080 log.cc:1196] T 31587501e8ab4fe88dcfd5b87ee60256 P 524dc62a7f9740b5b29c93bab94dafeb: Deleting WAL directory at /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/ts-1/wal/wals/31587501e8ab4fe88dcfd5b87ee60256
I20260928 06:33:51.009958 25080 ts_tablet_manager.cc:1939] T 31587501e8ab4fe88dcfd5b87ee60256 P 524dc62a7f9740b5b29c93bab94dafeb: Deleting consensus metadata
I20260928 06:33:51.013453 24529 catalog_manager.cc:4841] TS 524dc62a7f9740b5b29c93bab94dafeb (127.16.157.66:32923): tablet 31587501e8ab4fe88dcfd5b87ee60256 (table test-table [id=ff733fa3c82f4ddfa814e02056ba27f1]) successfully deleted
Sep 28 06:33:51 dist-test-slave-rrj5 krb5kdc[24497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 06:33:51 dist-test-slave-rrj5 krb5kdc[24497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:33:51 dist-test-slave-rrj5 krb5kdc[24497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 06:33:51.040071 24593 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:33:51.033152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51562 (local address 127.16.157.126:44663)
0928 06:33:51.033200 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:51.033203 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:51.033489 (+ 286us) server_negotiation.cc:407] Connection header received
0928 06:33:51.033588 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:51.033588 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:51.033600 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:51.033645 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:51.034373 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:51.035165 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:51.036287 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:51.036601 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:51.036644 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:51.039994 (+ 3350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51562: BlockingRecv error: recv got EOF from 127.0.0.1:51562 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 28 06:33:51 dist-test-slave-rrj5 krb5kdc[24497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM
I20260928 06:33:51.050261 24593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:51.041635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51576 (local address 127.16.157.126:44663)
0928 06:33:51.041669 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:51.041671 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:51.041844 (+ 173us) server_negotiation.cc:407] Connection header received
0928 06:33:51.041918 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:51.041919 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:51.041930 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:51.041976 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:51.042624 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:51.043376 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:51.044396 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:51.044694 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:51.044734 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:51.047224 (+ 2490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:51.047225 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:51.047225 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:51.047232 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:51.048918 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:51.049389 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:51.049390 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:51.049390 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:51.049461 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:51.049691 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:51.049691 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:51.049692 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:51.049818 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:51.049887 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:51.050097 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 06:33:51.050171 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 06:33:51.051997 24540 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51576:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:33:51.052171 24540 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:33:51.053738 24812 tablet_service.cc:1467] Processing CreateTablet for tablet 50fe67104e8f48d9900e381ad0e7314f (DEFAULT_TABLE table=test-table [id=8f4bfcecce5e476a95b3be85ebdb935b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:33:51.053799 24962 tablet_service.cc:1467] Processing CreateTablet for tablet 50fe67104e8f48d9900e381ad0e7314f (DEFAULT_TABLE table=test-table [id=8f4bfcecce5e476a95b3be85ebdb935b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:33:51.053890 24812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50fe67104e8f48d9900e381ad0e7314f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:51.053907 24962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50fe67104e8f48d9900e381ad0e7314f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:51.055223 25062 tablet_bootstrap.cc:492] T 50fe67104e8f48d9900e381ad0e7314f P 1940c10288ad429e8bee620822bcd4e5: Bootstrap starting.
I20260928 06:33:51.055224 25059 tablet_bootstrap.cc:492] T 50fe67104e8f48d9900e381ad0e7314f P 524dc62a7f9740b5b29c93bab94dafeb: Bootstrap starting.
I20260928 06:33:51.055548 25059 tablet_bootstrap.cc:654] T 50fe67104e8f48d9900e381ad0e7314f P 524dc62a7f9740b5b29c93bab94dafeb: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:51.055635 25062 tablet_bootstrap.cc:654] T 50fe67104e8f48d9900e381ad0e7314f P 1940c10288ad429e8bee620822bcd4e5: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:51.056283 24680 tablet_service.cc:1467] Processing CreateTablet for tablet 50fe67104e8f48d9900e381ad0e7314f (DEFAULT_TABLE table=test-table [id=8f4bfcecce5e476a95b3be85ebdb935b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:33:51.056421 24680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50fe67104e8f48d9900e381ad0e7314f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:51.057685 25058 tablet_bootstrap.cc:492] T 50fe67104e8f48d9900e381ad0e7314f P 32392e70db0445fe9341ff89e046a25d: Bootstrap starting.
I20260928 06:33:51.058010 25062 tablet_bootstrap.cc:492] T 50fe67104e8f48d9900e381ad0e7314f P 1940c10288ad429e8bee620822bcd4e5: No bootstrap required, opened a new log
I20260928 06:33:51.058066 25062 ts_tablet_manager.cc:1397] T 50fe67104e8f48d9900e381ad0e7314f P 1940c10288ad429e8bee620822bcd4e5: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260928 06:33:51.058178 25058 tablet_bootstrap.cc:654] T 50fe67104e8f48d9900e381ad0e7314f P 32392e70db0445fe9341ff89e046a25d: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:51.058228 25062 raft_consensus.cc:348] T 50fe67104e8f48d9900e381ad0e7314f P 1940c10288ad429e8bee620822bcd4e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32392e70db0445fe9341ff89e046a25d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 37273 } } peers { permanent_uuid: "524dc62a7f9740b5b29c93bab94dafeb" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32923 } } peers { permanent_uuid: "1940c10288ad429e8bee620822bcd4e5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39535 } }
I20260928 06:33:51.058321 25062 raft_consensus.cc:374] T 50fe67104e8f48d9900e381ad0e7314f P 1940c10288ad429e8bee620822bcd4e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:51.058343 25062 raft_consensus.cc:729] T 50fe67104e8f48d9900e381ad0e7314f P 1940c10288ad429e8bee620822bcd4e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1940c10288ad429e8bee620822bcd4e5, State: Initialized, Role: FOLLOWER
I20260928 06:33:51.058396 25062 consensus_queue.cc:260] T 50fe67104e8f48d9900e381ad0e7314f P 1940c10288ad429e8bee620822bcd4e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32392e70db0445fe9341ff89e046a25d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 37273 } } peers { permanent_uuid: "524dc62a7f9740b5b29c93bab94dafeb" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32923 } } peers { permanent_uuid: "1940c10288ad429e8bee620822bcd4e5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39535 } }
I20260928 06:33:51.058507 25062 ts_tablet_manager.cc:1428] T 50fe67104e8f48d9900e381ad0e7314f P 1940c10288ad429e8bee620822bcd4e5: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 06:33:51.066039 25058 tablet_bootstrap.cc:492] T 50fe67104e8f48d9900e381ad0e7314f P 32392e70db0445fe9341ff89e046a25d: No bootstrap required, opened a new log
I20260928 06:33:51.066112 25058 ts_tablet_manager.cc:1397] T 50fe67104e8f48d9900e381ad0e7314f P 32392e70db0445fe9341ff89e046a25d: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260928 06:33:51.066277 25058 raft_consensus.cc:348] T 50fe67104e8f48d9900e381ad0e7314f P 32392e70db0445fe9341ff89e046a25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32392e70db0445fe9341ff89e046a25d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 37273 } } peers { permanent_uuid: "524dc62a7f9740b5b29c93bab94dafeb" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32923 } } peers { permanent_uuid: "1940c10288ad429e8bee620822bcd4e5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39535 } }
I20260928 06:33:51.066357 25058 raft_consensus.cc:374] T 50fe67104e8f48d9900e381ad0e7314f P 32392e70db0445fe9341ff89e046a25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:51.066385 25058 raft_consensus.cc:729] T 50fe67104e8f48d9900e381ad0e7314f P 32392e70db0445fe9341ff89e046a25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32392e70db0445fe9341ff89e046a25d, State: Initialized, Role: FOLLOWER
I20260928 06:33:51.066426 25058 consensus_queue.cc:260] T 50fe67104e8f48d9900e381ad0e7314f P 32392e70db0445fe9341ff89e046a25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32392e70db0445fe9341ff89e046a25d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 37273 } } peers { permanent_uuid: "524dc62a7f9740b5b29c93bab94dafeb" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32923 } } peers { permanent_uuid: "1940c10288ad429e8bee620822bcd4e5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39535 } }
I20260928 06:33:51.066527 25058 ts_tablet_manager.cc:1428] T 50fe67104e8f48d9900e381ad0e7314f P 32392e70db0445fe9341ff89e046a25d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 06:33:51.067093 25059 tablet_bootstrap.cc:492] T 50fe67104e8f48d9900e381ad0e7314f P 524dc62a7f9740b5b29c93bab94dafeb: No bootstrap required, opened a new log
I20260928 06:33:51.067145 25059 ts_tablet_manager.cc:1397] T 50fe67104e8f48d9900e381ad0e7314f P 524dc62a7f9740b5b29c93bab94dafeb: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.001s
I20260928 06:33:51.067252 25059 raft_consensus.cc:348] T 50fe67104e8f48d9900e381ad0e7314f P 524dc62a7f9740b5b29c93bab94dafeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32392e70db0445fe9341ff89e046a25d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 37273 } } peers { permanent_uuid: "524dc62a7f9740b5b29c93bab94dafeb" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32923 } } peers { permanent_uuid: "1940c10288ad429e8bee620822bcd4e5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39535 } }
I20260928 06:33:51.067324 25059 raft_consensus.cc:374] T 50fe67104e8f48d9900e381ad0e7314f P 524dc62a7f9740b5b29c93bab94dafeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:51.067351 25059 raft_consensus.cc:729] T 50fe67104e8f48d9900e381ad0e7314f P 524dc62a7f9740b5b29c93bab94dafeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 524dc62a7f9740b5b29c93bab94dafeb, State: Initialized, Role: FOLLOWER
I20260928 06:33:51.067396 25059 consensus_queue.cc:260] T 50fe67104e8f48d9900e381ad0e7314f P 524dc62a7f9740b5b29c93bab94dafeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32392e70db0445fe9341ff89e046a25d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 37273 } } peers { permanent_uuid: "524dc62a7f9740b5b29c93bab94dafeb" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32923 } } peers { permanent_uuid: "1940c10288ad429e8bee620822bcd4e5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39535 } }
I20260928 06:33:51.067495 25059 ts_tablet_manager.cc:1428] T 50fe67104e8f48d9900e381ad0e7314f P 524dc62a7f9740b5b29c93bab94dafeb: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 06:33:51.241367 25066 raft_consensus.cc:482] T 50fe67104e8f48d9900e381ad0e7314f P 1940c10288ad429e8bee620822bcd4e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:33:51.241490 25066 raft_consensus.cc:504] T 50fe67104e8f48d9900e381ad0e7314f P 1940c10288ad429e8bee620822bcd4e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32392e70db0445fe9341ff89e046a25d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 37273 } } peers { permanent_uuid: "524dc62a7f9740b5b29c93bab94dafeb" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32923 } } peers { permanent_uuid: "1940c10288ad429e8bee620822bcd4e5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39535 } }
I20260928 06:33:51.241904 25066 leader_election.cc:288] T 50fe67104e8f48d9900e381ad0e7314f P 1940c10288ad429e8bee620822bcd4e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 32392e70db0445fe9341ff89e046a25d (127.16.157.65:37273), 524dc62a7f9740b5b29c93bab94dafeb (127.16.157.66:32923)
I20260928 06:33:51.245518 25054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:51.242213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:45001 (local address 127.16.157.65:37273)
0928 06:33:51.242302 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:51.242304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:51.242428 (+ 124us) server_negotiation.cc:407] Connection header received
0928 06:33:51.242516 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:51.242517 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:51.242531 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:51.242560 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:33:51.242801 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:51.243482 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:51.244618 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:51.245124 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:51.245196 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:51.245205 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:51.245399 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 06:33:51.245444 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 06:33:51.245599 25132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:51.242069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.65:37273 (local address 127.16.157.67:45001)
0928 06:33:51.242361 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:51.242372 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:51.242438 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:51.242628 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:51.242629 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:51.242631 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:33:51.242717 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:51.242719 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:51.243765 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:51.243765 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:51.244522 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:51.244524 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:51.245242 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:51.245244 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:51.245323 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:51.245328 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:33:51.245393 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 06:33:51.245444 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":17,"thread_start_us":54,"threads_started":1}
I20260928 06:33:51.245782 25055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:51.242199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:47997 (local address 127.16.157.66:32923)
0928 06:33:51.242264 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:51.242266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:51.242331 (+ 65us) server_negotiation.cc:407] Connection header received
0928 06:33:51.242401 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:51.242402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:51.242412 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:51.242466 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:33:51.242629 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:51.243392 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:51.244829 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:51.245351 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:51.245414 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:51.245420 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:51.245651 (+ 231us) server_negotiation.cc:299] Negotiation successful
0928 06:33:51.245703 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 06:33:51.245898 24701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fe67104e8f48d9900e381ad0e7314f" candidate_uuid: "1940c10288ad429e8bee620822bcd4e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32392e70db0445fe9341ff89e046a25d" is_pre_election: true
I20260928 06:33:51.245995 24701 raft_consensus.cc:2454] T 50fe67104e8f48d9900e381ad0e7314f P 32392e70db0445fe9341ff89e046a25d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1940c10288ad429e8bee620822bcd4e5 in term 0.
I20260928 06:33:51.246021 25134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:51.242069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.66:32923 (local address 127.16.157.67:47997)
0928 06:33:51.242259 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:51.242271 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:51.242344 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:51.242471 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:51.242472 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:51.242474 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:33:51.242550 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:51.242552 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:51.243489 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:51.243490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:51.244739 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:51.244742 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:51.245454 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:51.245455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:51.245527 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:51.245532 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:33:51.245903 (+ 371us) client_negotiation.cc:240] Negotiation successful
0928 06:33:51.245944 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":71,"threads_started":1}
I20260928 06:33:51.246241 24840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fe67104e8f48d9900e381ad0e7314f" candidate_uuid: "1940c10288ad429e8bee620822bcd4e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "524dc62a7f9740b5b29c93bab94dafeb" is_pre_election: true
I20260928 06:33:51.246321 24840 raft_consensus.cc:2454] T 50fe67104e8f48d9900e381ad0e7314f P 524dc62a7f9740b5b29c93bab94dafeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1940c10288ad429e8bee620822bcd4e5 in term 0.
I20260928 06:33:51.246294 24914 leader_election.cc:302] T 50fe67104e8f48d9900e381ad0e7314f P 1940c10288ad429e8bee620822bcd4e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1940c10288ad429e8bee620822bcd4e5, 32392e70db0445fe9341ff89e046a25d; no voters:
I20260928 06:33:51.246426 25066 raft_consensus.cc:2781] T 50fe67104e8f48d9900e381ad0e7314f P 1940c10288ad429e8bee620822bcd4e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:33:51.246470 25066 raft_consensus.cc:482] T 50fe67104e8f48d9900e381ad0e7314f P 1940c10288ad429e8bee620822bcd4e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:33:51.246495 25066 raft_consensus.cc:3037] T 50fe67104e8f48d9900e381ad0e7314f P 1940c10288ad429e8bee620822bcd4e5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:51.247494 25066 raft_consensus.cc:504] T 50fe67104e8f48d9900e381ad0e7314f P 1940c10288ad429e8bee620822bcd4e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32392e70db0445fe9341ff89e046a25d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 37273 } } peers { permanent_uuid: "524dc62a7f9740b5b29c93bab94dafeb" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32923 } } peers { permanent_uuid: "1940c10288ad429e8bee620822bcd4e5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39535 } }
I20260928 06:33:51.247639 25066 leader_election.cc:288] T 50fe67104e8f48d9900e381ad0e7314f P 1940c10288ad429e8bee620822bcd4e5 [CANDIDATE]: Term 1 election: Requested vote from peers 32392e70db0445fe9341ff89e046a25d (127.16.157.65:37273), 524dc62a7f9740b5b29c93bab94dafeb (127.16.157.66:32923)
I20260928 06:33:51.247809 24701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fe67104e8f48d9900e381ad0e7314f" candidate_uuid: "1940c10288ad429e8bee620822bcd4e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32392e70db0445fe9341ff89e046a25d"
I20260928 06:33:51.247898 24701 raft_consensus.cc:3037] T 50fe67104e8f48d9900e381ad0e7314f P 32392e70db0445fe9341ff89e046a25d [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:51.247890 24840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fe67104e8f48d9900e381ad0e7314f" candidate_uuid: "1940c10288ad429e8bee620822bcd4e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "524dc62a7f9740b5b29c93bab94dafeb"
I20260928 06:33:51.248009 24840 raft_consensus.cc:3037] T 50fe67104e8f48d9900e381ad0e7314f P 524dc62a7f9740b5b29c93bab94dafeb [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:51.248795 24701 raft_consensus.cc:2454] T 50fe67104e8f48d9900e381ad0e7314f P 32392e70db0445fe9341ff89e046a25d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1940c10288ad429e8bee620822bcd4e5 in term 1.
I20260928 06:33:51.248868 24840 raft_consensus.cc:2454] T 50fe67104e8f48d9900e381ad0e7314f P 524dc62a7f9740b5b29c93bab94dafeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1940c10288ad429e8bee620822bcd4e5 in term 1.
I20260928 06:33:51.248952 24914 leader_election.cc:302] T 50fe67104e8f48d9900e381ad0e7314f P 1940c10288ad429e8bee620822bcd4e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1940c10288ad429e8bee620822bcd4e5, 32392e70db0445fe9341ff89e046a25d; no voters:
I20260928 06:33:51.249023 25066 raft_consensus.cc:2781] T 50fe67104e8f48d9900e381ad0e7314f P 1940c10288ad429e8bee620822bcd4e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:33:51.249183 25066 raft_consensus.cc:686] T 50fe67104e8f48d9900e381ad0e7314f P 1940c10288ad429e8bee620822bcd4e5 [term 1 LEADER]: Becoming Leader. State: Replica: 1940c10288ad429e8bee620822bcd4e5, State: Running, Role: LEADER
I20260928 06:33:51.249259 25066 consensus_queue.cc:237] T 50fe67104e8f48d9900e381ad0e7314f P 1940c10288ad429e8bee620822bcd4e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32392e70db0445fe9341ff89e046a25d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 37273 } } peers { permanent_uuid: "524dc62a7f9740b5b29c93bab94dafeb" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32923 } } peers { permanent_uuid: "1940c10288ad429e8bee620822bcd4e5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39535 } }
I20260928 06:33:51.249994 24540 catalog_manager.cc:5495] T 50fe67104e8f48d9900e381ad0e7314f P 1940c10288ad429e8bee620822bcd4e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1940c10288ad429e8bee620822bcd4e5 (127.16.157.67). New cstate: current_term: 1 leader_uuid: "1940c10288ad429e8bee620822bcd4e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32392e70db0445fe9341ff89e046a25d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 37273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "524dc62a7f9740b5b29c93bab94dafeb" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1940c10288ad429e8bee620822bcd4e5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39535 } health_report { overall_health: HEALTHY } } }
I20260928 06:33:51.289777 25056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:51.286656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38878 (local address 127.16.157.67:39535)
0928 06:33:51.286725 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:51.286727 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:51.286736 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:33:51.286781 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:51.286782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:51.286804 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:51.286901 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:33:51.287039 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:51.287519 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:51.288761 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:51.289032 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:51.289121 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:51.289383 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:33:51.289510 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:33:51.289649 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:51.289676 (+ 27us) server_negotiation.cc:299] Negotiation successful
0928 06:33:51.289697 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 06:33:51.290897 24840 raft_consensus.cc:1261] T 50fe67104e8f48d9900e381ad0e7314f P 524dc62a7f9740b5b29c93bab94dafeb [term 1 FOLLOWER]: Refusing update from remote peer 1940c10288ad429e8bee620822bcd4e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:33:51.290905 24701 raft_consensus.cc:1261] T 50fe67104e8f48d9900e381ad0e7314f P 32392e70db0445fe9341ff89e046a25d [term 1 FOLLOWER]: Refusing update from remote peer 1940c10288ad429e8bee620822bcd4e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:33:51.291177 25066 consensus_queue.cc:1035] T 50fe67104e8f48d9900e381ad0e7314f P 1940c10288ad429e8bee620822bcd4e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "524dc62a7f9740b5b29c93bab94dafeb" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:33:51.291317 25135 consensus_queue.cc:1035] T 50fe67104e8f48d9900e381ad0e7314f P 1940c10288ad429e8bee620822bcd4e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32392e70db0445fe9341ff89e046a25d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 37273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:33:51.295046 24540 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:33:51.295141 24540 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:33:51.295815 24540 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 217c529c34ff472987667c1af6f76a6b: Sending DeleteTablet for 3 replicas of tablet 50fe67104e8f48d9900e381ad0e7314f
I20260928 06:33:51.296096 24812 tablet_service.cc:1514] Processing DeleteTablet for tablet 50fe67104e8f48d9900e381ad0e7314f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:33:51 UTC) from {username='oryx', principal='oryx/127.16.157.126@KRBTEST.COM'} at 127.0.0.1:42992
I20260928 06:33:51.296241 25080 tablet_replica.cc:330] T 50fe67104e8f48d9900e381ad0e7314f P 524dc62a7f9740b5b29c93bab94dafeb: stopping tablet replica
I20260928 06:33:51.296326 25080 raft_consensus.cc:2229] T 50fe67104e8f48d9900e381ad0e7314f P 524dc62a7f9740b5b29c93bab94dafeb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:33:51.296373 25080 raft_consensus.cc:2258] T 50fe67104e8f48d9900e381ad0e7314f P 524dc62a7f9740b5b29c93bab94dafeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:33:51.296355 24962 tablet_service.cc:1514] Processing DeleteTablet for tablet 50fe67104e8f48d9900e381ad0e7314f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:33:51 UTC) from {username='oryx', principal='oryx/127.16.157.126@KRBTEST.COM'} at 127.0.0.1:38864
I20260928 06:33:51.296471 25082 tablet_replica.cc:330] T 50fe67104e8f48d9900e381ad0e7314f P 1940c10288ad429e8bee620822bcd4e5: stopping tablet replica
I20260928 06:33:51.296527 25082 raft_consensus.cc:2229] T 50fe67104e8f48d9900e381ad0e7314f P 1940c10288ad429e8bee620822bcd4e5 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:33:51.296613 25082 raft_consensus.cc:2258] T 50fe67104e8f48d9900e381ad0e7314f P 1940c10288ad429e8bee620822bcd4e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:33:51.296747 24680 tablet_service.cc:1514] Processing DeleteTablet for tablet 50fe67104e8f48d9900e381ad0e7314f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:33:51 UTC) from {username='oryx', principal='oryx/127.16.157.126@KRBTEST.COM'} at 127.0.0.1:34288
I20260928 06:33:51.296859 25081 tablet_replica.cc:330] T 50fe67104e8f48d9900e381ad0e7314f P 32392e70db0445fe9341ff89e046a25d: stopping tablet replica
I20260928 06:33:51.296928 25081 raft_consensus.cc:2229] T 50fe67104e8f48d9900e381ad0e7314f P 32392e70db0445fe9341ff89e046a25d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:33:51.296981 25081 raft_consensus.cc:2258] T 50fe67104e8f48d9900e381ad0e7314f P 32392e70db0445fe9341ff89e046a25d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:33:51.297089 24540 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51576:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 06:33:51.297199 25080 ts_tablet_manager.cc:1905] T 50fe67104e8f48d9900e381ad0e7314f P 524dc62a7f9740b5b29c93bab94dafeb: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 06:33:51.297235 24540 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:33:51.297245 25081 ts_tablet_manager.cc:1905] T 50fe67104e8f48d9900e381ad0e7314f P 32392e70db0445fe9341ff89e046a25d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:33:51.298941 25081 ts_tablet_manager.cc:1918] T 50fe67104e8f48d9900e381ad0e7314f P 32392e70db0445fe9341ff89e046a25d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:33:51.298967 25080 ts_tablet_manager.cc:1918] T 50fe67104e8f48d9900e381ad0e7314f P 524dc62a7f9740b5b29c93bab94dafeb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:33:51.299002 25081 log.cc:1196] T 50fe67104e8f48d9900e381ad0e7314f P 32392e70db0445fe9341ff89e046a25d: Deleting WAL directory at /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/ts-0/wal/wals/50fe67104e8f48d9900e381ad0e7314f
I20260928 06:33:51.299022 25080 log.cc:1196] T 50fe67104e8f48d9900e381ad0e7314f P 524dc62a7f9740b5b29c93bab94dafeb: Deleting WAL directory at /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/ts-1/wal/wals/50fe67104e8f48d9900e381ad0e7314f
I20260928 06:33:51.299243 25081 ts_tablet_manager.cc:1939] T 50fe67104e8f48d9900e381ad0e7314f P 32392e70db0445fe9341ff89e046a25d: Deleting consensus metadata
I20260928 06:33:51.299278 25080 ts_tablet_manager.cc:1939] T 50fe67104e8f48d9900e381ad0e7314f P 524dc62a7f9740b5b29c93bab94dafeb: Deleting consensus metadata
I20260928 06:33:51.299525 24812 tablet_service.cc:1467] Processing CreateTablet for tablet af7cf1b328b94cacb0fed2122cb15e5a (DEFAULT_TABLE table=test-table [id=caf3aabfeff04338958f7aaf7496a680]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:33:51.299635 24812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af7cf1b328b94cacb0fed2122cb15e5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:51.300796 25059 tablet_bootstrap.cc:492] T af7cf1b328b94cacb0fed2122cb15e5a P 524dc62a7f9740b5b29c93bab94dafeb: Bootstrap starting.
I20260928 06:33:51.300869 24529 catalog_manager.cc:4841] TS 524dc62a7f9740b5b29c93bab94dafeb (127.16.157.66:32923): tablet 50fe67104e8f48d9900e381ad0e7314f (table test-table [id=8f4bfcecce5e476a95b3be85ebdb935b]) successfully deleted
I20260928 06:33:51.301105 25059 tablet_bootstrap.cc:654] T af7cf1b328b94cacb0fed2122cb15e5a P 524dc62a7f9740b5b29c93bab94dafeb: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:51.301113 24962 tablet_service.cc:1467] Processing CreateTablet for tablet af7cf1b328b94cacb0fed2122cb15e5a (DEFAULT_TABLE table=test-table [id=caf3aabfeff04338958f7aaf7496a680]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:33:51.301229 24962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af7cf1b328b94cacb0fed2122cb15e5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:51.301681 25059 tablet_bootstrap.cc:492] T af7cf1b328b94cacb0fed2122cb15e5a P 524dc62a7f9740b5b29c93bab94dafeb: No bootstrap required, opened a new log
I20260928 06:33:51.301728 25059 ts_tablet_manager.cc:1397] T af7cf1b328b94cacb0fed2122cb15e5a P 524dc62a7f9740b5b29c93bab94dafeb: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 06:33:51.301834 25059 raft_consensus.cc:348] T af7cf1b328b94cacb0fed2122cb15e5a P 524dc62a7f9740b5b29c93bab94dafeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32392e70db0445fe9341ff89e046a25d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 37273 } } peers { permanent_uuid: "524dc62a7f9740b5b29c93bab94dafeb" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32923 } } peers { permanent_uuid: "1940c10288ad429e8bee620822bcd4e5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39535 } }
I20260928 06:33:51.301895 25059 raft_consensus.cc:374] T af7cf1b328b94cacb0fed2122cb15e5a P 524dc62a7f9740b5b29c93bab94dafeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:51.301916 25059 raft_consensus.cc:729] T af7cf1b328b94cacb0fed2122cb15e5a P 524dc62a7f9740b5b29c93bab94dafeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 524dc62a7f9740b5b29c93bab94dafeb, State: Initialized, Role: FOLLOWER
I20260928 06:33:51.301959 25059 consensus_queue.cc:260] T af7cf1b328b94cacb0fed2122cb15e5a P 524dc62a7f9740b5b29c93bab94dafeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32392e70db0445fe9341ff89e046a25d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 37273 } } peers { permanent_uuid: "524dc62a7f9740b5b29c93bab94dafeb" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32923 } } peers { permanent_uuid: "1940c10288ad429e8bee620822bcd4e5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39535 } }
I20260928 06:33:51.302042 25059 ts_tablet_manager.cc:1428] T af7cf1b328b94cacb0fed2122cb15e5a P 524dc62a7f9740b5b29c93bab94dafeb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:51.302670 25062 tablet_bootstrap.cc:492] T af7cf1b328b94cacb0fed2122cb15e5a P 1940c10288ad429e8bee620822bcd4e5: Bootstrap starting.
I20260928 06:33:51.302975 24528 catalog_manager.cc:4841] TS 32392e70db0445fe9341ff89e046a25d (127.16.157.65:37273): tablet 50fe67104e8f48d9900e381ad0e7314f (table test-table [id=8f4bfcecce5e476a95b3be85ebdb935b]) successfully deleted
I20260928 06:33:51.303074 25062 tablet_bootstrap.cc:654] T af7cf1b328b94cacb0fed2122cb15e5a P 1940c10288ad429e8bee620822bcd4e5: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:51.303398 24680 tablet_service.cc:1467] Processing CreateTablet for tablet af7cf1b328b94cacb0fed2122cb15e5a (DEFAULT_TABLE table=test-table [id=caf3aabfeff04338958f7aaf7496a680]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:33:51.303505 24680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af7cf1b328b94cacb0fed2122cb15e5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:51.304131 25082 ts_tablet_manager.cc:1905] T 50fe67104e8f48d9900e381ad0e7314f P 1940c10288ad429e8bee620822bcd4e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:33:51.304718 25058 tablet_bootstrap.cc:492] T af7cf1b328b94cacb0fed2122cb15e5a P 32392e70db0445fe9341ff89e046a25d: Bootstrap starting.
I20260928 06:33:51.305001 25058 tablet_bootstrap.cc:654] T af7cf1b328b94cacb0fed2122cb15e5a P 32392e70db0445fe9341ff89e046a25d: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:51.305469 25062 tablet_bootstrap.cc:492] T af7cf1b328b94cacb0fed2122cb15e5a P 1940c10288ad429e8bee620822bcd4e5: No bootstrap required, opened a new log
I20260928 06:33:51.305516 25062 ts_tablet_manager.cc:1397] T af7cf1b328b94cacb0fed2122cb15e5a P 1940c10288ad429e8bee620822bcd4e5: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260928 06:33:51.305662 25082 ts_tablet_manager.cc:1918] T 50fe67104e8f48d9900e381ad0e7314f P 1940c10288ad429e8bee620822bcd4e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:33:51.305730 25082 log.cc:1196] T 50fe67104e8f48d9900e381ad0e7314f P 1940c10288ad429e8bee620822bcd4e5: Deleting WAL directory at /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/ts-2/wal/wals/50fe67104e8f48d9900e381ad0e7314f
I20260928 06:33:51.305958 25082 ts_tablet_manager.cc:1939] T 50fe67104e8f48d9900e381ad0e7314f P 1940c10288ad429e8bee620822bcd4e5: Deleting consensus metadata
I20260928 06:33:51.306109 25058 tablet_bootstrap.cc:492] T af7cf1b328b94cacb0fed2122cb15e5a P 32392e70db0445fe9341ff89e046a25d: No bootstrap required, opened a new log
I20260928 06:33:51.306151 25058 ts_tablet_manager.cc:1397] T af7cf1b328b94cacb0fed2122cb15e5a P 32392e70db0445fe9341ff89e046a25d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:51.306249 25058 raft_consensus.cc:348] T af7cf1b328b94cacb0fed2122cb15e5a P 32392e70db0445fe9341ff89e046a25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32392e70db0445fe9341ff89e046a25d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 37273 } } peers { permanent_uuid: "524dc62a7f9740b5b29c93bab94dafeb" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32923 } } peers { permanent_uuid: "1940c10288ad429e8bee620822bcd4e5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39535 } }
I20260928 06:33:51.306306 25058 raft_consensus.cc:374] T af7cf1b328b94cacb0fed2122cb15e5a P 32392e70db0445fe9341ff89e046a25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:51.306325 25058 raft_consensus.cc:729] T af7cf1b328b94cacb0fed2122cb15e5a P 32392e70db0445fe9341ff89e046a25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32392e70db0445fe9341ff89e046a25d, State: Initialized, Role: FOLLOWER
I20260928 06:33:51.306322 24526 catalog_manager.cc:4841] TS 1940c10288ad429e8bee620822bcd4e5 (127.16.157.67:39535): tablet 50fe67104e8f48d9900e381ad0e7314f (table test-table [id=8f4bfcecce5e476a95b3be85ebdb935b]) successfully deleted
I20260928 06:33:51.306366 25058 consensus_queue.cc:260] T af7cf1b328b94cacb0fed2122cb15e5a P 32392e70db0445fe9341ff89e046a25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32392e70db0445fe9341ff89e046a25d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 37273 } } peers { permanent_uuid: "524dc62a7f9740b5b29c93bab94dafeb" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32923 } } peers { permanent_uuid: "1940c10288ad429e8bee620822bcd4e5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39535 } }
I20260928 06:33:51.306460 25058 ts_tablet_manager.cc:1428] T af7cf1b328b94cacb0fed2122cb15e5a P 32392e70db0445fe9341ff89e046a25d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:51.305665 25062 raft_consensus.cc:348] T af7cf1b328b94cacb0fed2122cb15e5a P 1940c10288ad429e8bee620822bcd4e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32392e70db0445fe9341ff89e046a25d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 37273 } } peers { permanent_uuid: "524dc62a7f9740b5b29c93bab94dafeb" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32923 } } peers { permanent_uuid: "1940c10288ad429e8bee620822bcd4e5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39535 } }
I20260928 06:33:51.306663 25062 raft_consensus.cc:374] T af7cf1b328b94cacb0fed2122cb15e5a P 1940c10288ad429e8bee620822bcd4e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:51.306691 25062 raft_consensus.cc:729] T af7cf1b328b94cacb0fed2122cb15e5a P 1940c10288ad429e8bee620822bcd4e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1940c10288ad429e8bee620822bcd4e5, State: Initialized, Role: FOLLOWER
I20260928 06:33:51.306737 25062 consensus_queue.cc:260] T af7cf1b328b94cacb0fed2122cb15e5a P 1940c10288ad429e8bee620822bcd4e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32392e70db0445fe9341ff89e046a25d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 37273 } } peers { permanent_uuid: "524dc62a7f9740b5b29c93bab94dafeb" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32923 } } peers { permanent_uuid: "1940c10288ad429e8bee620822bcd4e5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39535 } }
I20260928 06:33:51.306862 25062 ts_tablet_manager.cc:1428] T af7cf1b328b94cacb0fed2122cb15e5a P 1940c10288ad429e8bee620822bcd4e5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 06:33:51.312207 25140 raft_consensus.cc:482] T af7cf1b328b94cacb0fed2122cb15e5a P 1940c10288ad429e8bee620822bcd4e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:33:51.312273 25140 raft_consensus.cc:504] T af7cf1b328b94cacb0fed2122cb15e5a P 1940c10288ad429e8bee620822bcd4e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32392e70db0445fe9341ff89e046a25d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 37273 } } peers { permanent_uuid: "524dc62a7f9740b5b29c93bab94dafeb" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32923 } } peers { permanent_uuid: "1940c10288ad429e8bee620822bcd4e5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39535 } }
I20260928 06:33:51.312428 25140 leader_election.cc:288] T af7cf1b328b94cacb0fed2122cb15e5a P 1940c10288ad429e8bee620822bcd4e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 32392e70db0445fe9341ff89e046a25d (127.16.157.65:37273), 524dc62a7f9740b5b29c93bab94dafeb (127.16.157.66:32923)
I20260928 06:33:51.312564 24701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af7cf1b328b94cacb0fed2122cb15e5a" candidate_uuid: "1940c10288ad429e8bee620822bcd4e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32392e70db0445fe9341ff89e046a25d" is_pre_election: true
I20260928 06:33:51.312659 24701 raft_consensus.cc:2454] T af7cf1b328b94cacb0fed2122cb15e5a P 32392e70db0445fe9341ff89e046a25d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1940c10288ad429e8bee620822bcd4e5 in term 0.
I20260928 06:33:51.312702 24840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af7cf1b328b94cacb0fed2122cb15e5a" candidate_uuid: "1940c10288ad429e8bee620822bcd4e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "524dc62a7f9740b5b29c93bab94dafeb" is_pre_election: true
I20260928 06:33:51.312791 24840 raft_consensus.cc:2454] T af7cf1b328b94cacb0fed2122cb15e5a P 524dc62a7f9740b5b29c93bab94dafeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1940c10288ad429e8bee620822bcd4e5 in term 0.
I20260928 06:33:51.312939 24915 leader_election.cc:302] T af7cf1b328b94cacb0fed2122cb15e5a P 1940c10288ad429e8bee620822bcd4e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1940c10288ad429e8bee620822bcd4e5, 524dc62a7f9740b5b29c93bab94dafeb; no voters:
I20260928 06:33:51.313059 25140 raft_consensus.cc:2781] T af7cf1b328b94cacb0fed2122cb15e5a P 1940c10288ad429e8bee620822bcd4e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:33:51.313128 25140 raft_consensus.cc:482] T af7cf1b328b94cacb0fed2122cb15e5a P 1940c10288ad429e8bee620822bcd4e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:33:51.313192 25140 raft_consensus.cc:3037] T af7cf1b328b94cacb0fed2122cb15e5a P 1940c10288ad429e8bee620822bcd4e5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:51.314006 25140 raft_consensus.cc:504] T af7cf1b328b94cacb0fed2122cb15e5a P 1940c10288ad429e8bee620822bcd4e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32392e70db0445fe9341ff89e046a25d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 37273 } } peers { permanent_uuid: "524dc62a7f9740b5b29c93bab94dafeb" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32923 } } peers { permanent_uuid: "1940c10288ad429e8bee620822bcd4e5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39535 } }
I20260928 06:33:51.314245 25140 leader_election.cc:288] T af7cf1b328b94cacb0fed2122cb15e5a P 1940c10288ad429e8bee620822bcd4e5 [CANDIDATE]: Term 1 election: Requested vote from peers 32392e70db0445fe9341ff89e046a25d (127.16.157.65:37273), 524dc62a7f9740b5b29c93bab94dafeb (127.16.157.66:32923)
I20260928 06:33:51.314417 24701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af7cf1b328b94cacb0fed2122cb15e5a" candidate_uuid: "1940c10288ad429e8bee620822bcd4e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32392e70db0445fe9341ff89e046a25d"
I20260928 06:33:51.314489 24701 raft_consensus.cc:3037] T af7cf1b328b94cacb0fed2122cb15e5a P 32392e70db0445fe9341ff89e046a25d [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:51.314476 24840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af7cf1b328b94cacb0fed2122cb15e5a" candidate_uuid: "1940c10288ad429e8bee620822bcd4e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "524dc62a7f9740b5b29c93bab94dafeb"
I20260928 06:33:51.314534 24840 raft_consensus.cc:3037] T af7cf1b328b94cacb0fed2122cb15e5a P 524dc62a7f9740b5b29c93bab94dafeb [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:51.315165 24701 raft_consensus.cc:2454] T af7cf1b328b94cacb0fed2122cb15e5a P 32392e70db0445fe9341ff89e046a25d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1940c10288ad429e8bee620822bcd4e5 in term 1.
I20260928 06:33:51.315220 24840 raft_consensus.cc:2454] T af7cf1b328b94cacb0fed2122cb15e5a P 524dc62a7f9740b5b29c93bab94dafeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1940c10288ad429e8bee620822bcd4e5 in term 1.
I20260928 06:33:51.315440 24915 leader_election.cc:302] T af7cf1b328b94cacb0fed2122cb15e5a P 1940c10288ad429e8bee620822bcd4e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1940c10288ad429e8bee620822bcd4e5, 524dc62a7f9740b5b29c93bab94dafeb; no voters:
I20260928 06:33:51.315578 25140 raft_consensus.cc:2781] T af7cf1b328b94cacb0fed2122cb15e5a P 1940c10288ad429e8bee620822bcd4e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:33:51.315819 25140 raft_consensus.cc:686] T af7cf1b328b94cacb0fed2122cb15e5a P 1940c10288ad429e8bee620822bcd4e5 [term 1 LEADER]: Becoming Leader. State: Replica: 1940c10288ad429e8bee620822bcd4e5, State: Running, Role: LEADER
I20260928 06:33:51.315907 25140 consensus_queue.cc:237] T af7cf1b328b94cacb0fed2122cb15e5a P 1940c10288ad429e8bee620822bcd4e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32392e70db0445fe9341ff89e046a25d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 37273 } } peers { permanent_uuid: "524dc62a7f9740b5b29c93bab94dafeb" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32923 } } peers { permanent_uuid: "1940c10288ad429e8bee620822bcd4e5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39535 } }
I20260928 06:33:51.316393 24540 catalog_manager.cc:5495] T af7cf1b328b94cacb0fed2122cb15e5a P 1940c10288ad429e8bee620822bcd4e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1940c10288ad429e8bee620822bcd4e5 (127.16.157.67). New cstate: current_term: 1 leader_uuid: "1940c10288ad429e8bee620822bcd4e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32392e70db0445fe9341ff89e046a25d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 37273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "524dc62a7f9740b5b29c93bab94dafeb" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1940c10288ad429e8bee620822bcd4e5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39535 } health_report { overall_health: HEALTHY } } }
I20260928 06:33:51.327777 25055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:51.325387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:45251 (local address 127.16.157.66:32923)
0928 06:33:51.325434 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:51.325436 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:51.325444 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:33:51.325498 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:51.325499 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:51.325509 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:51.325564 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:33:51.325767 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:51.326251 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:51.327035 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:51.327255 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:51.327341 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:51.327524 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:33:51.327556 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:33:51.327627 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:51.327688 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 06:33:51.327710 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 06:33:51.327819 25148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:51.325215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.66:32923 (local address 127.16.157.67:45251)
0928 06:33:51.325375 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:51.325388 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:51.325443 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:51.325595 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:51.325596 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:51.325597 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:33:51.325665 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:51.325667 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:51.326333 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:51.326334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:51.326949 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:51.326951 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:51.327343 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:51.327344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:51.327407 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:51.327429 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:33:51.327629 (+ 200us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:33:51.327631 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:33:51.327683 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 06:33:51.327732 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":96,"threads_started":1}
I20260928 06:33:51.329658 24701 raft_consensus.cc:1261] T af7cf1b328b94cacb0fed2122cb15e5a P 32392e70db0445fe9341ff89e046a25d [term 1 FOLLOWER]: Refusing update from remote peer 1940c10288ad429e8bee620822bcd4e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:33:51.329649 24840 raft_consensus.cc:1261] T af7cf1b328b94cacb0fed2122cb15e5a P 524dc62a7f9740b5b29c93bab94dafeb [term 1 FOLLOWER]: Refusing update from remote peer 1940c10288ad429e8bee620822bcd4e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:33:51.329846 25140 consensus_queue.cc:1035] T af7cf1b328b94cacb0fed2122cb15e5a P 1940c10288ad429e8bee620822bcd4e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "524dc62a7f9740b5b29c93bab94dafeb" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:33:51.329972 25135 consensus_queue.cc:1035] T af7cf1b328b94cacb0fed2122cb15e5a P 1940c10288ad429e8bee620822bcd4e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32392e70db0445fe9341ff89e046a25d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 37273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:33:51.337872 25056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:51.334721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.66:40417 (local address 127.16.157.67:39535)
0928 06:33:51.334760 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:51.334762 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:51.334771 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:33:51.334894 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:51.334895 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:51.334904 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:51.334947 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:33:51.335090 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:51.335843 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:51.336987 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:51.337345 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:51.337423 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:51.337612 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:33:51.337652 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:33:51.337719 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:51.337776 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 06:33:51.337796 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:33:51.337904 25111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:51.334623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.67:39535 (local address 127.16.157.66:40417)
0928 06:33:51.334732 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:51.334743 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:51.334809 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:51.334984 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:51.334985 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:51.334987 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:33:51.335023 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:51.335025 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:51.335959 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:51.335960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:51.336888 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:51.336891 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:51.337432 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:51.337433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:51.337499 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:51.337506 (+ 7us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:33:51.337722 (+ 216us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:33:51.337724 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:33:51.337773 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 06:33:51.337824 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260928 06:33:51.346066 24540 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:33:51.346171 24540 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:33:51.346771 24540 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 217c529c34ff472987667c1af6f76a6b: Sending DeleteTablet for 3 replicas of tablet af7cf1b328b94cacb0fed2122cb15e5a
I20260928 06:33:51.347045 24812 tablet_service.cc:1514] Processing DeleteTablet for tablet af7cf1b328b94cacb0fed2122cb15e5a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:33:51 UTC) from {username='oryx', principal='oryx/127.16.157.126@KRBTEST.COM'} at 127.0.0.1:42992
I20260928 06:33:51.347132 25080 tablet_replica.cc:330] T af7cf1b328b94cacb0fed2122cb15e5a P 524dc62a7f9740b5b29c93bab94dafeb: stopping tablet replica
I20260928 06:33:51.347185 25080 raft_consensus.cc:2229] T af7cf1b328b94cacb0fed2122cb15e5a P 524dc62a7f9740b5b29c93bab94dafeb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:33:51.347221 25080 raft_consensus.cc:2258] T af7cf1b328b94cacb0fed2122cb15e5a P 524dc62a7f9740b5b29c93bab94dafeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:33:51.347350 24962 tablet_service.cc:1514] Processing DeleteTablet for tablet af7cf1b328b94cacb0fed2122cb15e5a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:33:51 UTC) from {username='oryx', principal='oryx/127.16.157.126@KRBTEST.COM'} at 127.0.0.1:38864
I20260928 06:33:51.347393 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 24615
I20260928 06:33:51.347427 25082 tablet_replica.cc:330] T af7cf1b328b94cacb0fed2122cb15e5a P 1940c10288ad429e8bee620822bcd4e5: stopping tablet replica
I20260928 06:33:51.347478 25082 raft_consensus.cc:2229] T af7cf1b328b94cacb0fed2122cb15e5a P 1940c10288ad429e8bee620822bcd4e5 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:33:51.347532 25082 raft_consensus.cc:2258] T af7cf1b328b94cacb0fed2122cb15e5a P 1940c10288ad429e8bee620822bcd4e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:33:51.348281 25082 ts_tablet_manager.cc:1905] T af7cf1b328b94cacb0fed2122cb15e5a P 1940c10288ad429e8bee620822bcd4e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:33:51.349972 25082 ts_tablet_manager.cc:1918] T af7cf1b328b94cacb0fed2122cb15e5a P 1940c10288ad429e8bee620822bcd4e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:33:51.350044 25082 log.cc:1196] T af7cf1b328b94cacb0fed2122cb15e5a P 1940c10288ad429e8bee620822bcd4e5: Deleting WAL directory at /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/ts-2/wal/wals/af7cf1b328b94cacb0fed2122cb15e5a
I20260928 06:33:51.350265 25082 ts_tablet_manager.cc:1939] T af7cf1b328b94cacb0fed2122cb15e5a P 1940c10288ad429e8bee620822bcd4e5: Deleting consensus metadata
I20260928 06:33:51.350625 24526 catalog_manager.cc:4841] TS 1940c10288ad429e8bee620822bcd4e5 (127.16.157.67:39535): tablet af7cf1b328b94cacb0fed2122cb15e5a (table test-table [id=caf3aabfeff04338958f7aaf7496a680]) successfully deleted
I20260928 06:33:51.354121 25080 ts_tablet_manager.cc:1905] T af7cf1b328b94cacb0fed2122cb15e5a P 524dc62a7f9740b5b29c93bab94dafeb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:33:51.355443 25080 ts_tablet_manager.cc:1918] T af7cf1b328b94cacb0fed2122cb15e5a P 524dc62a7f9740b5b29c93bab94dafeb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:33:51.355523 25080 log.cc:1196] T af7cf1b328b94cacb0fed2122cb15e5a P 524dc62a7f9740b5b29c93bab94dafeb: Deleting WAL directory at /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577196946045-17013-0/minicluster-data/ts-1/wal/wals/af7cf1b328b94cacb0fed2122cb15e5a
W20260928 06:33:51.355739 24528 connection.cc:535] client connection to 127.16.157.65:37273 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 06:33:51.355788 25080 ts_tablet_manager.cc:1939] T af7cf1b328b94cacb0fed2122cb15e5a P 524dc62a7f9740b5b29c93bab94dafeb: Deleting consensus metadata
W20260928 06:33:51.355875 24528 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 06:33:51.356384 25152 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:33:51.356183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.65:37273 (local address 127.0.0.1:34304)
0928 06:33:51.356284 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:51.356336 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.65:37273: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":80,"thread_start_us":42,"threads_started":1}
I20260928 06:33:51.356429 24529 catalog_manager.cc:4841] TS 524dc62a7f9740b5b29c93bab94dafeb (127.16.157.66:32923): tablet af7cf1b328b94cacb0fed2122cb15e5a (table test-table [id=caf3aabfeff04338958f7aaf7496a680]) successfully deleted
W20260928 06:33:51.356482 24528 catalog_manager.cc:4584] TS 32392e70db0445fe9341ff89e046a25d (127.16.157.65:37273): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet af7cf1b328b94cacb0fed2122cb15e5a: Network error: Client connection negotiation failed: client connection to 127.16.157.65:37273: connect: Connection reset by peer (error 104)
I20260928 06:33:51.356711 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 24756
I20260928 06:33:51.366273 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 24896
I20260928 06:33:51.371165 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 24510
2026-09-28T06:33:51Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3051 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577196946045-17013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:33:51 dist-test-slave-rrj5 krb5kdc[25155](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:33:51 dist-test-slave-rrj5 krb5kdc[25155](info): set up 2 sockets
Sep 28 06:33:51 dist-test-slave-rrj5 krb5kdc[25155](info): commencing operation
krb5kdc: starting...
W20260928 06:33:52.427078 17013 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Sep 28 06:33:52 dist-test-slave-rrj5 krb5kdc[25155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:33:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:33:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.157.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab.
Entry for principal kudu/127.16.157.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab.
I20260928 06:33:52.548192 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577196946045-17013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577196946045-17013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577196946045-17013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:33195
--webserver_interface=127.16.157.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=kudu/127.16.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:33333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.157.126:33195
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:52.626899 25168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:52.627084 25168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:52.627104 25168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:52.628180 25168 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:33:52.628217 25168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:52.628232 25168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:52.628243 25168 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:33:52.628255 25168 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:33:52.629679 25168 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:33333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577196946045-17013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577196946045-17013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.157.126:33195
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:33195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577196946045-17013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:52.629904 25168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:52.630141 25168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:52.634837 25174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:52.634969 25173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:52.635018 25168 server_base.cc:1034] running on GCE node
W20260928 06:33:52.634853 25176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:52.635285 25168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:52.635598 25168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:52.636755 25168 hybrid_clock.cc:648] HybridClock initialized: now 1790577232636749 us; error 32 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T06:33:52Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1272 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:33:52 dist-test-slave-rrj5 krb5kdc[25181](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:33:52 dist-test-slave-rrj5 krb5kdc[25181](info): set up 2 sockets
Sep 28 06:33:52 dist-test-slave-rrj5 krb5kdc[25181](info): commencing operation
krb5kdc: starting...
W20260928 06:33:53.691138 17013 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Sep 28 06:33:53 dist-test-slave-rrj5 krb5kdc[25181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:33:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:33:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.157.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab.
Entry for principal kudu/127.16.157.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab.
I20260928 06:33:53.816561 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:40521
--webserver_interface=127.16.157.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=kudu/127.16.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:37789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.157.126:40521
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:53.900020 25194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:53.900213 25194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:53.900235 25194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:53.901422 25194 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:33:53.901465 25194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:53.901479 25194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:53.901494 25194 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:33:53.901505 25194 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:33:53.902951 25194 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:37789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.157.126:40521
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=kudu/127.16.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:40521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:53.903191 25194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:53.903404 25194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:53.908290 25199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:53.908331 25200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:53.908327 25202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:53.908576 25194 server_base.cc:1034] running on GCE node
I20260928 06:33:53.908809 25194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:53.909092 25194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:53.910259 25194 hybrid_clock.cc:648] HybridClock initialized: now 1790577233910249 us; error 36 us; skew 500 ppm
Sep 28 06:33:53 dist-test-slave-rrj5 krb5kdc[25181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577233, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:53.912809 25194 init.cc:377] Logged in from keytab as kudu/127.16.157.126@KRBTEST.COM (short username kudu)
I20260928 06:33:53.913287 25194 webserver.cc:460] Webserver started at http://127.16.157.126:38743/ using document root <none> and password file <none>
I20260928 06:33:53.913512 25194 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:53.913599 25194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:53.913756 25194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:53.914806 25194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/master-0/data/instance:
uuid: "22fc71853e744b1f831a82e2f4ea55f3"
format_stamp: "Formatted at 2026-09-28 06:33:53 on dist-test-slave-rrj5"
server_key: "853ea579cdfa47f20677deebc55db763"
server_key_iv: "22a2d275a722549e433d9aff74f77efa"
server_key_version: "encryptionkey@0"
I20260928 06:33:53.915181 25194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/master-0/wal/instance:
uuid: "22fc71853e744b1f831a82e2f4ea55f3"
format_stamp: "Formatted at 2026-09-28 06:33:53 on dist-test-slave-rrj5"
server_key: "853ea579cdfa47f20677deebc55db763"
server_key_iv: "22a2d275a722549e433d9aff74f77efa"
server_key_version: "encryptionkey@0"
I20260928 06:33:53.916581 25194 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:33:53.917409 25209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:53.917620 25194 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:53.917721 25194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/master-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/master-0/wal
uuid: "22fc71853e744b1f831a82e2f4ea55f3"
format_stamp: "Formatted at 2026-09-28 06:33:53 on dist-test-slave-rrj5"
server_key: "853ea579cdfa47f20677deebc55db763"
server_key_iv: "22a2d275a722549e433d9aff74f77efa"
server_key_version: "encryptionkey@0"
I20260928 06:33:53.917809 25194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:53.930004 25194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:53.932103 25194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:53.932283 25194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:53.937240 25194 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.126:40521
I20260928 06:33:53.937266 25261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.126:40521 every 8 connection(s)
I20260928 06:33:53.937736 25194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
I20260928 06:33:53.938724 25262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:53.941131 25262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22fc71853e744b1f831a82e2f4ea55f3: Bootstrap starting.
I20260928 06:33:53.941207 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 25194
I20260928 06:33:53.941311 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/master-0/wal/instance
I20260928 06:33:53.941502 17013 external_mini_cluster.cc:1428] Setting key af148f53e7d06dd82c5df4c1ef779d49
I20260928 06:33:53.941887 25262 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22fc71853e744b1f831a82e2f4ea55f3: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:53.942237 25262 log.cc:824] T 00000000000000000000000000000000 P 22fc71853e744b1f831a82e2f4ea55f3: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:53.942962 25262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22fc71853e744b1f831a82e2f4ea55f3: No bootstrap required, opened a new log
I20260928 06:33:53.944846 25262 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22fc71853e744b1f831a82e2f4ea55f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22fc71853e744b1f831a82e2f4ea55f3" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40521 } }
I20260928 06:33:53.945011 25262 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22fc71853e744b1f831a82e2f4ea55f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:53.945061 25262 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22fc71853e744b1f831a82e2f4ea55f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22fc71853e744b1f831a82e2f4ea55f3, State: Initialized, Role: FOLLOWER
I20260928 06:33:53.945217 25262 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22fc71853e744b1f831a82e2f4ea55f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22fc71853e744b1f831a82e2f4ea55f3" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40521 } }
I20260928 06:33:53.945312 25262 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22fc71853e744b1f831a82e2f4ea55f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:33:53.945356 25262 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22fc71853e744b1f831a82e2f4ea55f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:33:53.945402 25262 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22fc71853e744b1f831a82e2f4ea55f3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:53.946128 25262 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22fc71853e744b1f831a82e2f4ea55f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22fc71853e744b1f831a82e2f4ea55f3" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40521 } }
I20260928 06:33:53.946244 25262 leader_election.cc:302] T 00000000000000000000000000000000 P 22fc71853e744b1f831a82e2f4ea55f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22fc71853e744b1f831a82e2f4ea55f3; no voters:
I20260928 06:33:53.946417 25262 leader_election.cc:288] T 00000000000000000000000000000000 P 22fc71853e744b1f831a82e2f4ea55f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:33:53.946523 25267 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22fc71853e744b1f831a82e2f4ea55f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:33:53.946686 25262 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22fc71853e744b1f831a82e2f4ea55f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:33:53.946708 25267 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22fc71853e744b1f831a82e2f4ea55f3 [term 1 LEADER]: Becoming Leader. State: Replica: 22fc71853e744b1f831a82e2f4ea55f3, State: Running, Role: LEADER
I20260928 06:33:53.946892 25267 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22fc71853e744b1f831a82e2f4ea55f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22fc71853e744b1f831a82e2f4ea55f3" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40521 } }
I20260928 06:33:53.947265 25268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22fc71853e744b1f831a82e2f4ea55f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22fc71853e744b1f831a82e2f4ea55f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22fc71853e744b1f831a82e2f4ea55f3" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40521 } } }
I20260928 06:33:53.947343 25268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22fc71853e744b1f831a82e2f4ea55f3 [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:53.947520 25269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22fc71853e744b1f831a82e2f4ea55f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22fc71853e744b1f831a82e2f4ea55f3. Latest consensus state: current_term: 1 leader_uuid: "22fc71853e744b1f831a82e2f4ea55f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22fc71853e744b1f831a82e2f4ea55f3" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40521 } } }
I20260928 06:33:53.947572 25269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22fc71853e744b1f831a82e2f4ea55f3 [sys.catalog]: This master's current role is: LEADER
W20260928 06:33:53.948665 25284 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 22fc71853e744b1f831a82e2f4ea55f3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:33:53.948761 25284 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:33:53.948849 25277 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:33:53.949018 25277 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 06:33:53 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:33:53.950893 25277 catalog_manager.cc:1347] Generated new cluster ID: cada3cdee8ce446e9d06348f234a93fa
I20260928 06:33:53.950937 25277 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:33:53.955216 25265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:53.941991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50394 (local address 127.16.157.126:40521)
0928 06:33:53.942256 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:53.942265 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:53.942298 (+ 33us) server_negotiation.cc:407] Connection header received
0928 06:33:53.942884 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:53.942893 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:53.942966 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:53.943887 (+ 921us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:53.943983 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:53.945380 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:53.946578 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:53.947215 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:53.947317 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:53.950806 (+ 3489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:53.950810 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:53.950811 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:53.950822 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:53.953030 (+ 2208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:53.953493 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:53.953496 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:53.953497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:53.953634 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:53.953892 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:53.953893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:53.953893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:53.954207 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:53.954371 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:53.954857 (+ 486us) server_negotiation.cc:299] Negotiation successful
0928 06:33:53.954999 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":85,"threads_started":1}
I20260928 06:33:53.962203 25277 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:33:53.963184 25277 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:33:53.976594 25277 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22fc71853e744b1f831a82e2f4ea55f3: Generated new TSK 0
I20260928 06:33:53.976776 25277 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.157.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab.
Entry for principal kudu/127.16.157.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab.
I20260928 06:33:54.008049 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.65:0
--local_ip_for_outbound_sockets=127.16.157.65
--webserver_interface=127.16.157.65
--webserver_port=0
--tserver_master_addrs=127.16.157.126:40521
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab
--principal=kudu/127.16.157.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:37789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:54.092572 25289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:54.092794 25289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:54.092834 25289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:54.094072 25289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:54.094144 25289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:54.094193 25289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.65
I20260928 06:33:54.095624 25289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:37789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab
--principal=kudu/127.16.157.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.157.65
--webserver_port=0
--tserver_master_addrs=127.16.157.126:40521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.65
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:54.095888 25289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:54.096143 25289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:54.101135 25295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:54.101157 25297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:54.102035 25294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:54.102380 25289 server_base.cc:1034] running on GCE node
I20260928 06:33:54.102653 25289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:54.102933 25289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:54.104099 25289 hybrid_clock.cc:648] HybridClock initialized: now 1790577234104071 us; error 47 us; skew 500 ppm
Sep 28 06:33:54 dist-test-slave-rrj5 krb5kdc[25181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577234, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:54.106882 25289 init.cc:377] Logged in from keytab as kudu/127.16.157.65@KRBTEST.COM (short username kudu)
I20260928 06:33:54.107431 25289 webserver.cc:460] Webserver started at http://127.16.157.65:46785/ using document root <none> and password file <none>
I20260928 06:33:54.107683 25289 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:54.107757 25289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:54.107915 25289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:54.109026 25289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-0/data/instance:
uuid: "5a46648f7eed412695f1cb5f51e7f919"
format_stamp: "Formatted at 2026-09-28 06:33:54 on dist-test-slave-rrj5"
server_key: "899fe73db2772b35bd321df4259371e0"
server_key_iv: "f6ea1c71d58276416c01b0ba12845dfc"
server_key_version: "encryptionkey@0"
I20260928 06:33:54.109418 25289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-0/wal/instance:
uuid: "5a46648f7eed412695f1cb5f51e7f919"
format_stamp: "Formatted at 2026-09-28 06:33:54 on dist-test-slave-rrj5"
server_key: "899fe73db2772b35bd321df4259371e0"
server_key_iv: "f6ea1c71d58276416c01b0ba12845dfc"
server_key_version: "encryptionkey@0"
I20260928 06:33:54.110934 25289 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:54.111773 25304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:54.111954 25289 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:54.112043 25289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-0/wal
uuid: "5a46648f7eed412695f1cb5f51e7f919"
format_stamp: "Formatted at 2026-09-28 06:33:54 on dist-test-slave-rrj5"
server_key: "899fe73db2772b35bd321df4259371e0"
server_key_iv: "f6ea1c71d58276416c01b0ba12845dfc"
server_key_version: "encryptionkey@0"
I20260928 06:33:54.112149 25289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:54.131853 25289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:54.134256 25289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:54.134428 25289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:54.134670 25289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:33:54.134982 25289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:33:54.135041 25289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:54.135089 25289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:33:54.135133 25289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:54.141873 25289 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.65:42147
I20260928 06:33:54.141971 25417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.65:42147 every 8 connection(s)
I20260928 06:33:54.142288 25289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
I20260928 06:33:54.143397 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 25289
I20260928 06:33:54.143508 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-0/wal/instance
I20260928 06:33:54.143726 17013 external_mini_cluster.cc:1428] Setting key a3b5cd17985d011f971837de0fb95bca
Sep 28 06:33:54 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577234, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.65@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.157.66@KRBTEST.COM; defaulting to no policy
I20260928 06:33:54.154984 25265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:54.143050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:47379 (local address 127.16.157.126:40521)
0928 06:33:54.143125 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:54.143127 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:54.143323 (+ 196us) server_negotiation.cc:407] Connection header received
0928 06:33:54.144387 (+ 1064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:54.144388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:54.144400 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:54.144468 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:54.145759 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:54.146543 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:54.148116 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:54.148457 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:54.148505 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:54.151334 (+ 2829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:54.151336 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:54.151336 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:54.151345 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:54.153145 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:54.153672 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:54.153673 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:54.153674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:54.153754 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:54.154154 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:54.154155 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:54.154156 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:54.154316 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:54.154395 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:54.154765 (+ 370us) server_negotiation.cc:299] Negotiation successful
0928 06:33:54.154865 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 06:33:54.154989 25421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:54.142997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40521 (local address 127.16.157.65:47379)
0928 06:33:54.143177 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:54.143220 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:54.143953 (+ 733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:54.144597 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:54.144602 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:54.145351 (+ 749us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:54.145641 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:54.145647 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:54.146684 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:54.146685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:54.148006 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:54.148009 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:54.148599 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:54.148600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:54.148748 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:54.149303 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:54.149313 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:54.151225 (+ 1912us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:54.153247 (+ 2022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:54.153253 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:54.153255 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:54.153537 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:54.153920 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:54.153921 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:54.153922 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:54.154040 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:54.154425 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:54.154431 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:54.154600 (+ 169us) client_negotiation.cc:769] Sending connection context
0928 06:33:54.154689 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 06:33:54.154806 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":87,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.66@KRBTEST.COM" created.
I20260928 06:33:54.155522 25419 heartbeater.cc:347] Connected to a master server at 127.16.157.126:40521
I20260928 06:33:54.155618 25419 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:54.155804 25419 heartbeater.cc:511] Master 127.16.157.126:40521 requested a full tablet report, sending...
I20260928 06:33:54.156381 25226 ts_manager.cc:194] Registered new tserver with Master: 5a46648f7eed412695f1cb5f51e7f919 (127.16.157.65:42147)
I20260928 06:33:54.157091 25226 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.157.65@KRBTEST.COM'} at 127.16.157.65:47379
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab.
Entry for principal kudu/127.16.157.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab.
I20260928 06:33:54.170712 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.66:0
--local_ip_for_outbound_sockets=127.16.157.66
--webserver_interface=127.16.157.66
--webserver_port=0
--tserver_master_addrs=127.16.157.126:40521
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab
--principal=kudu/127.16.157.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:37789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:54.256337 25424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:54.256518 25424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:54.256541 25424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:54.257762 25424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:54.257803 25424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:54.257830 25424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.66
I20260928 06:33:54.259233 25424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:37789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab
--principal=kudu/127.16.157.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.157.66
--webserver_port=0
--tserver_master_addrs=127.16.157.126:40521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.66
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:54.259456 25424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:54.259689 25424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:54.264760 25432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:54.264824 25430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:54.264907 25424 server_base.cc:1034] running on GCE node
W20260928 06:33:54.264961 25429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:54.265247 25424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:54.265512 25424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:54.266680 25424 hybrid_clock.cc:648] HybridClock initialized: now 1790577234266661 us; error 35 us; skew 500 ppm
Sep 28 06:33:54 dist-test-slave-rrj5 krb5kdc[25181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577234, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:54.269101 25424 init.cc:377] Logged in from keytab as kudu/127.16.157.66@KRBTEST.COM (short username kudu)
I20260928 06:33:54.269605 25424 webserver.cc:460] Webserver started at http://127.16.157.66:39653/ using document root <none> and password file <none>
I20260928 06:33:54.269841 25424 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:54.269912 25424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:54.270072 25424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:54.271116 25424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-1/data/instance:
uuid: "452c125941c846538f27dc86e08e7281"
format_stamp: "Formatted at 2026-09-28 06:33:54 on dist-test-slave-rrj5"
server_key: "6372b410dea5538904b6378db79c0248"
server_key_iv: "192f78624023fc979a2e9f2c0a7f4daa"
server_key_version: "encryptionkey@0"
I20260928 06:33:54.271484 25424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-1/wal/instance:
uuid: "452c125941c846538f27dc86e08e7281"
format_stamp: "Formatted at 2026-09-28 06:33:54 on dist-test-slave-rrj5"
server_key: "6372b410dea5538904b6378db79c0248"
server_key_iv: "192f78624023fc979a2e9f2c0a7f4daa"
server_key_version: "encryptionkey@0"
I20260928 06:33:54.272867 25424 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:33:54.273659 25439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:54.273849 25424 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:54.273952 25424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-1/wal
uuid: "452c125941c846538f27dc86e08e7281"
format_stamp: "Formatted at 2026-09-28 06:33:54 on dist-test-slave-rrj5"
server_key: "6372b410dea5538904b6378db79c0248"
server_key_iv: "192f78624023fc979a2e9f2c0a7f4daa"
server_key_version: "encryptionkey@0"
I20260928 06:33:54.274039 25424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:54.288537 25424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:54.291190 25424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:54.291386 25424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:54.291682 25424 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:33:54.292116 25424 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:33:54.292161 25424 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:54.292227 25424 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:33:54.292258 25424 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:54.299683 25424 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.66:39763
I20260928 06:33:54.299726 25552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.66:39763 every 8 connection(s)
I20260928 06:33:54.300034 25424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
I20260928 06:33:54.305544 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 25424
I20260928 06:33:54.305680 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-1/wal/instance
I20260928 06:33:54.305879 17013 external_mini_cluster.cc:1428] Setting key 49589e3af48f79a32e9c1da79db62862
Sep 28 06:33:54 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577234, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.66@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:33:54.310320 25265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:54.300806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.66:58535 (local address 127.16.157.126:40521)
0928 06:33:54.300889 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:54.300891 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:54.301080 (+ 189us) server_negotiation.cc:407] Connection header received
0928 06:33:54.301620 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:54.301622 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:54.301634 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:54.301681 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:54.302582 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:54.303104 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:54.304529 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:54.304725 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:54.304847 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:54.307229 (+ 2382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:54.307230 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:54.307231 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:54.307239 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:54.308796 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:54.309216 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:54.309218 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:54.309218 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:54.309276 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:54.309536 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:54.309537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:54.309537 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:54.309691 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:54.309785 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:54.310141 (+ 356us) server_negotiation.cc:299] Negotiation successful
0928 06:33:54.310228 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 06:33:54.310678 25555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:54.300796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40521 (local address 127.16.157.66:58535)
0928 06:33:54.300956 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:54.300996 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:54.301500 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:54.301762 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:54.301765 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:54.302293 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:54.302479 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:54.302484 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:54.303205 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:54.303206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:54.304437 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:54.304441 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:54.304855 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:54.304856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:54.304997 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:54.305490 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:54.305499 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:54.307130 (+ 1631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:54.308906 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:54.308913 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:54.308915 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:54.309137 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:54.309350 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:54.309351 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:54.309352 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:54.309467 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:54.309818 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:54.309826 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:54.310000 (+ 174us) client_negotiation.cc:769] Sending connection context
0928 06:33:54.310119 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 06:33:54.310272 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":89,"threads_started":1}
I20260928 06:33:54.311199 25553 heartbeater.cc:347] Connected to a master server at 127.16.157.126:40521
I20260928 06:33:54.311306 25553 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:54.311483 25553 heartbeater.cc:511] Master 127.16.157.126:40521 requested a full tablet report, sending...
I20260928 06:33:54.311910 25226 ts_manager.cc:194] Registered new tserver with Master: 452c125941c846538f27dc86e08e7281 (127.16.157.66:39763)
I20260928 06:33:54.312440 25226 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.157.66@KRBTEST.COM'} at 127.16.157.66:58535
WARNING: no policy specified for kudu/127.16.157.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab.
Entry for principal kudu/127.16.157.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab.
I20260928 06:33:54.333169 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.67:0
--local_ip_for_outbound_sockets=127.16.157.67
--webserver_interface=127.16.157.67
--webserver_port=0
--tserver_master_addrs=127.16.157.126:40521
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab
--principal=kudu/127.16.157.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:37789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:54.436352 25559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:54.436547 25559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:54.436573 25559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:54.437799 25559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:54.437839 25559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:54.437866 25559 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.67
I20260928 06:33:54.439297 25559 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:37789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab
--principal=kudu/127.16.157.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.157.67
--webserver_port=0
--tserver_master_addrs=127.16.157.126:40521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25559
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.67
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:54.439527 25559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:54.439751 25559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:54.444832 25565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:54.444936 25564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:54.444857 25567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:54.444968 25559 server_base.cc:1034] running on GCE node
I20260928 06:33:54.445137 25559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:54.445431 25559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:54.446581 25559 hybrid_clock.cc:648] HybridClock initialized: now 1790577234446565 us; error 34 us; skew 500 ppm
Sep 28 06:33:54 dist-test-slave-rrj5 krb5kdc[25181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577234, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:54.449404 25559 init.cc:377] Logged in from keytab as kudu/127.16.157.67@KRBTEST.COM (short username kudu)
I20260928 06:33:54.449879 25559 webserver.cc:460] Webserver started at http://127.16.157.67:45725/ using document root <none> and password file <none>
I20260928 06:33:54.450102 25559 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:54.450145 25559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:54.450316 25559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:54.451345 25559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-2/data/instance:
uuid: "1c863190a3ff41f48cbf546ff5486728"
format_stamp: "Formatted at 2026-09-28 06:33:54 on dist-test-slave-rrj5"
server_key: "926e14a0f025c5135797d5d0232db2ca"
server_key_iv: "6ecc269a33ce73bd5dfaa128b686a12d"
server_key_version: "encryptionkey@0"
I20260928 06:33:54.451676 25559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-2/wal/instance:
uuid: "1c863190a3ff41f48cbf546ff5486728"
format_stamp: "Formatted at 2026-09-28 06:33:54 on dist-test-slave-rrj5"
server_key: "926e14a0f025c5135797d5d0232db2ca"
server_key_iv: "6ecc269a33ce73bd5dfaa128b686a12d"
server_key_version: "encryptionkey@0"
I20260928 06:33:54.453043 25559 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:33:54.453859 25574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:54.454054 25559 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:33:54.454152 25559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-2/wal
uuid: "1c863190a3ff41f48cbf546ff5486728"
format_stamp: "Formatted at 2026-09-28 06:33:54 on dist-test-slave-rrj5"
server_key: "926e14a0f025c5135797d5d0232db2ca"
server_key_iv: "6ecc269a33ce73bd5dfaa128b686a12d"
server_key_version: "encryptionkey@0"
I20260928 06:33:54.454247 25559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:54.485266 25559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:54.487835 25559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:54.488003 25559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:54.488284 25559 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:33:54.488654 25559 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:33:54.488699 25559 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:54.488729 25559 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:33:54.488755 25559 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:54.496291 25559 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.67:38307
I20260928 06:33:54.496366 25687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.67:38307 every 8 connection(s)
I20260928 06:33:54.496711 25559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
I20260928 06:33:54.498245 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 25559
I20260928 06:33:54.498351 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-2/wal/instance
I20260928 06:33:54.498546 17013 external_mini_cluster.cc:1428] Setting key b8443e8ada0fef397dbdfffa090798e0
Sep 28 06:33:54 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577234, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.67@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:33:54.507225 25265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:54.497469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:53129 (local address 127.16.157.126:40521)
0928 06:33:54.497543 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:54.497545 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:54.497841 (+ 296us) server_negotiation.cc:407] Connection header received
0928 06:33:54.498482 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:54.498483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:54.498499 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:54.498571 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:54.499788 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:54.500292 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:54.501659 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:54.501867 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:54.501921 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:54.504190 (+ 2269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:54.504192 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:54.504192 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:54.504200 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:54.505795 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:54.506235 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:54.506237 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:54.506237 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:54.506301 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:54.506571 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:54.506572 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:54.506572 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:54.506688 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:54.506764 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:54.507052 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 06:33:54.507132 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 06:33:54.507444 25690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:54.497440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40521 (local address 127.16.157.67:53129)
0928 06:33:54.497677 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:54.497728 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:54.498341 (+ 613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:54.498644 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:54.498648 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:54.499418 (+ 770us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:54.499695 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:54.499703 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:54.500397 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:54.500398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:54.501542 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:54.501544 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:54.501948 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:54.501949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:54.502082 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:54.502592 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:54.502602 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:54.504082 (+ 1480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:54.505910 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:54.505917 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:54.505920 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:54.506155 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:54.506384 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:54.506385 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:54.506385 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:54.506495 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:54.506755 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:54.506762 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:54.506908 (+ 146us) client_negotiation.cc:769] Sending connection context
0928 06:33:54.507017 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 06:33:54.507185 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":142,"threads_started":1}
I20260928 06:33:54.507956 25688 heartbeater.cc:347] Connected to a master server at 127.16.157.126:40521
I20260928 06:33:54.508059 25688 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:54.508283 25688 heartbeater.cc:511] Master 127.16.157.126:40521 requested a full tablet report, sending...
I20260928 06:33:54.508723 25226 ts_manager.cc:194] Registered new tserver with Master: 1c863190a3ff41f48cbf546ff5486728 (127.16.157.67:38307)
I20260928 06:33:54.509186 25226 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.157.67@KRBTEST.COM'} at 127.16.157.67:53129
I20260928 06:33:54.510006 17013 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.16.157.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.157.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.157.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/krb5kdc/oryx_127.16.157.65.keytab.
Entry for principal oryx/127.16.157.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/krb5kdc/oryx_127.16.157.65.keytab.
WARNING: no policy specified for oryx/127.16.157.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.157.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.157.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/krb5kdc/oryx_127.16.157.66.keytab.
Entry for principal oryx/127.16.157.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/krb5kdc/oryx_127.16.157.66.keytab.
WARNING: no policy specified for oryx/127.16.157.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.157.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.157.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/krb5kdc/oryx_127.16.157.67.keytab.
Entry for principal oryx/127.16.157.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/krb5kdc/oryx_127.16.157.67.keytab.
I20260928 06:33:54.588044 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 25289
I20260928 06:33:54.593426 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 25424
I20260928 06:33:54.598088 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 25559
I20260928 06:33:54.603523 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 25194
I20260928 06:33:54.609181 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:40521
--webserver_interface=127.16.157.126
--webserver_port=38743
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=kudu/127.16.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:37789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.157.126:40521
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:54.713070 25697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:54.713295 25697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:54.713333 25697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:54.714830 25697 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:33:54.714922 25697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:54.714948 25697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:54.714977 25697 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:33:54.715003 25697 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:33:54.716574 25697 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:37789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.157.126:40521
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=kudu/127.16.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:40521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.157.126
--webserver_port=38743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:54.716846 25697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:54.717139 25697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:54.722286 25703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:54.722311 25705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:54.722319 25702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:54.724314 25697 server_base.cc:1034] running on GCE node
I20260928 06:33:54.724633 25697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:54.725026 25697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:54.726202 25697 hybrid_clock.cc:648] HybridClock initialized: now 1790577234726163 us; error 55 us; skew 500 ppm
Sep 28 06:33:54 dist-test-slave-rrj5 krb5kdc[25181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577234, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:54.729481 25697 init.cc:377] Logged in from keytab as kudu/127.16.157.126@KRBTEST.COM (short username kudu)
I20260928 06:33:54.730032 25697 webserver.cc:460] Webserver started at http://127.16.157.126:38743/ using document root <none> and password file <none>
I20260928 06:33:54.730319 25697 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:54.730397 25697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:54.732195 25697 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:33:54.732997 25712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:54.733160 25697 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:54.733256 25697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/master-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/master-0/wal
uuid: "22fc71853e744b1f831a82e2f4ea55f3"
format_stamp: "Formatted at 2026-09-28 06:33:53 on dist-test-slave-rrj5"
server_key: "853ea579cdfa47f20677deebc55db763"
server_key_iv: "22a2d275a722549e433d9aff74f77efa"
server_key_version: "encryptionkey@0"
I20260928 06:33:54.733708 25697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:54.762912 25697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:54.765551 25697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:54.765725 25697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:54.770507 25697 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.126:40521
I20260928 06:33:54.770573 25764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.126:40521 every 8 connection(s)
I20260928 06:33:54.770959 25697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
I20260928 06:33:54.772388 25765 sys_catalog.cc:263] Verifying existing consensus state
I20260928 06:33:54.773206 25765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22fc71853e744b1f831a82e2f4ea55f3: Bootstrap starting.
I20260928 06:33:54.774681 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 25697
I20260928 06:33:54.775017 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.65:42147
--local_ip_for_outbound_sockets=127.16.157.65
--tserver_master_addrs=127.16.157.126:40521
--webserver_port=46785
--webserver_interface=127.16.157.65
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab
--principal=kudu/127.16.157.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:37789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.16.157.65
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/krb5kdc/oryx_127.16.157.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:33:54.782193 25765 log.cc:824] T 00000000000000000000000000000000 P 22fc71853e744b1f831a82e2f4ea55f3: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:54.784143 25765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22fc71853e744b1f831a82e2f4ea55f3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:33:54.784482 25765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22fc71853e744b1f831a82e2f4ea55f3: Bootstrap complete.
I20260928 06:33:54.787635 25765 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22fc71853e744b1f831a82e2f4ea55f3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22fc71853e744b1f831a82e2f4ea55f3" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40521 } }
I20260928 06:33:54.787983 25765 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22fc71853e744b1f831a82e2f4ea55f3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22fc71853e744b1f831a82e2f4ea55f3, State: Initialized, Role: FOLLOWER
I20260928 06:33:54.788158 25765 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22fc71853e744b1f831a82e2f4ea55f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22fc71853e744b1f831a82e2f4ea55f3" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40521 } }
I20260928 06:33:54.788228 25765 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22fc71853e744b1f831a82e2f4ea55f3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:33:54.788285 25765 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22fc71853e744b1f831a82e2f4ea55f3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:33:54.788326 25765 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22fc71853e744b1f831a82e2f4ea55f3 [term 1 FOLLOWER]: Advancing to term 2
I20260928 06:33:54.789553 25765 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22fc71853e744b1f831a82e2f4ea55f3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22fc71853e744b1f831a82e2f4ea55f3" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40521 } }
I20260928 06:33:54.789691 25765 leader_election.cc:302] T 00000000000000000000000000000000 P 22fc71853e744b1f831a82e2f4ea55f3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22fc71853e744b1f831a82e2f4ea55f3; no voters:
I20260928 06:33:54.789844 25765 leader_election.cc:288] T 00000000000000000000000000000000 P 22fc71853e744b1f831a82e2f4ea55f3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 06:33:54.789945 25769 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22fc71853e744b1f831a82e2f4ea55f3 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 06:33:54.790091 25765 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22fc71853e744b1f831a82e2f4ea55f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:33:54.790238 25769 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22fc71853e744b1f831a82e2f4ea55f3 [term 2 LEADER]: Becoming Leader. State: Replica: 22fc71853e744b1f831a82e2f4ea55f3, State: Running, Role: LEADER
I20260928 06:33:54.790390 25769 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22fc71853e744b1f831a82e2f4ea55f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22fc71853e744b1f831a82e2f4ea55f3" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40521 } }
I20260928 06:33:54.790706 25770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22fc71853e744b1f831a82e2f4ea55f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "22fc71853e744b1f831a82e2f4ea55f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22fc71853e744b1f831a82e2f4ea55f3" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40521 } } }
I20260928 06:33:54.790773 25770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22fc71853e744b1f831a82e2f4ea55f3 [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:54.790889 25770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22fc71853e744b1f831a82e2f4ea55f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22fc71853e744b1f831a82e2f4ea55f3. Latest consensus state: current_term: 2 leader_uuid: "22fc71853e744b1f831a82e2f4ea55f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22fc71853e744b1f831a82e2f4ea55f3" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40521 } } }
I20260928 06:33:54.790928 25770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22fc71853e744b1f831a82e2f4ea55f3 [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:54.792042 25786 catalog_manager.cc:1259] Loaded cluster ID: cada3cdee8ce446e9d06348f234a93fa
I20260928 06:33:54.792129 25786 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 22fc71853e744b1f831a82e2f4ea55f3: loading cluster ID for follower catalog manager: success
I20260928 06:33:54.793431 25786 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 22fc71853e744b1f831a82e2f4ea55f3: acquiring CA information for follower catalog manager: success
I20260928 06:33:54.794124 25786 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 22fc71853e744b1f831a82e2f4ea55f3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 06:33:54.797660 25778 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:33:54.797895 25778 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:33:54.798019 25778 catalog_manager.cc:1259] Loaded cluster ID: cada3cdee8ce446e9d06348f234a93fa
I20260928 06:33:54.798072 25778 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:33:54.798278 25778 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:33:54.798405 25778 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 22fc71853e744b1f831a82e2f4ea55f3: Loaded TSK: 0
I20260928 06:33:54.798573 25778 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 06:33:54.895911 25767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:54.896195 25767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:54.896235 25767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:54.898000 25767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:54.898056 25767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:54.898101 25767 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.65
I20260928 06:33:54.900259 25767 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:37789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/krb5kdc/oryx_127.16.157.65.keytab
--principal=oryx/127.16.157.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.65:42147
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.157.65
--webserver_port=46785
--tserver_master_addrs=127.16.157.126:40521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.65
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:54.900524 25767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:54.900779 25767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:54.906877 25793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:54.906862 25792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:54.909714 25795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:54.918949 25767 server_base.cc:1034] running on GCE node
I20260928 06:33:54.919200 25767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:54.919498 25767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:54.920673 25767 hybrid_clock.cc:648] HybridClock initialized: now 1790577234920653 us; error 35 us; skew 500 ppm
Sep 28 06:33:54 dist-test-slave-rrj5 krb5kdc[25181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577234, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.157.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:54.923974 25767 init.cc:377] Logged in from keytab as oryx/127.16.157.65@KRBTEST.COM (short username oryx)
I20260928 06:33:54.924523 25767 webserver.cc:460] Webserver started at http://127.16.157.65:46785/ using document root <none> and password file <none>
I20260928 06:33:54.924794 25767 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:54.924870 25767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:54.926581 25767 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:33:54.927218 25802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:54.927390 25767 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:33:54.927472 25767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-0/wal
uuid: "5a46648f7eed412695f1cb5f51e7f919"
format_stamp: "Formatted at 2026-09-28 06:33:54 on dist-test-slave-rrj5"
server_key: "899fe73db2772b35bd321df4259371e0"
server_key_iv: "f6ea1c71d58276416c01b0ba12845dfc"
server_key_version: "encryptionkey@0"
I20260928 06:33:54.927932 25767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:54.953833 25767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:54.956615 25767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:54.956816 25767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:54.957100 25767 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:33:54.957485 25767 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:33:54.957602 25767 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:54.957705 25767 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:33:54.957778 25767 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:54.965013 25767 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.65:42147
I20260928 06:33:54.965492 25767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
I20260928 06:33:54.971261 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 25767
I20260928 06:33:54.971589 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.66:39763
--local_ip_for_outbound_sockets=127.16.157.66
--tserver_master_addrs=127.16.157.126:40521
--webserver_port=39653
--webserver_interface=127.16.157.66
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab
--principal=kudu/127.16.157.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:37789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.16.157.66
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/krb5kdc/oryx_127.16.157.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:33:54.973766 25915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.65:42147 every 8 connection(s)
Sep 28 06:33:54 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.65@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:33:54 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.65@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 06:33:54.992420 25919 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:33:54.966369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:43365 (local address 127.16.157.126:40521)
0928 06:33:54.966558 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:54.966567 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:54.981930 (+ 15363us) server_negotiation.cc:407] Connection header received
0928 06:33:54.982839 (+ 909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:54.982848 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:54.982911 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:54.983029 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:54.984217 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:54.985524 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:54.987226 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:54.987917 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:54.988005 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:54.992222 (+ 4217us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.157.65:43365: BlockingRecv error: recv got EOF from 127.16.157.65:43365 (error 108)
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":85,"threads_started":1}
I20260928 06:33:54.992830 25918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:54.966321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40521 (local address 127.16.157.65:43365)
0928 06:33:54.981713 (+ 15392us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:54.981780 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:54.982563 (+ 783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:54.983258 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:54.983262 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:54.983878 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:54.984074 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:54.984079 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:54.985711 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:54.985713 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:54.987102 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:54.987105 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:54.988197 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:54.988198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:54.988351 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:54.989005 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:54.989015 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:54.992629 (+ 3614us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.157.126:40521: Server oryx/127.16.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":15331,"thread_start_us":92,"threads_started":1}
W20260928 06:33:54.993072 25916 heartbeater.cc:644] Failed to heartbeat to 127.16.157.126:40521 (0 consecutive failures): Not authorized: Failed to ping master at 127.16.157.126:40521: Client connection negotiation failed: client connection to 127.16.157.126:40521: Server oryx/127.16.157.126@KRBTEST.COM not found in Kerberos database
Sep 28 06:33:54 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.65@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:33:54 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.65@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:33:54.998627 25918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:54.993342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40521 (local address 127.16.157.65:60739)
0928 06:33:54.993514 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:54.993524 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:54.993604 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:54.993774 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:54.993774 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:54.993915 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:54.993986 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:54.993988 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:54.994765 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:54.994767 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:54.995517 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:54.995519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:54.996009 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:54.996010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:54.996091 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:54.996557 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:54.996563 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:54.998542 (+ 1979us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.157.126:40521: Server oryx/127.16.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159}
W20260928 06:33:54.998688 25919 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:33:54.993448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:60739 (local address 127.16.157.126:40521)
0928 06:33:54.993488 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:54.993490 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:54.993564 (+ 74us) server_negotiation.cc:407] Connection header received
0928 06:33:54.993692 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:54.993693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:54.993706 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:54.993772 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:54.994064 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:54.994682 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:54.995606 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:54.995857 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:54.995952 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:54.998626 (+ 2674us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.157.65:60739: BlockingRecv error: recv got EOF from 127.16.157.65:60739 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
Sep 28 06:33:55 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.65@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:33:55 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.65@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 06:33:55.004323 25919 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:33:54.999023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:57639 (local address 127.16.157.126:40521)
0928 06:33:54.999086 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:54.999087 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:54.999097 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:33:54.999196 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:54.999197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:54.999209 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:54.999241 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:54.999580 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:55.000097 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:55.001148 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:55.001450 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:55.001514 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:55.004266 (+ 2752us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.157.65:57639: BlockingRecv error: recv got EOF from 127.16.157.65:57639 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 06:33:55.004534 25918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:54.998938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40521 (local address 127.16.157.65:57639)
0928 06:33:54.999053 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:54.999062 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:54.999103 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:54.999302 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:54.999303 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:54.999453 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:54.999484 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:54.999485 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:55.000230 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:55.000231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:55.001043 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:55.001045 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:55.001619 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:55.001620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:55.001694 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:55.002172 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:55.002177 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:55.004470 (+ 2293us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.157.126:40521: Server oryx/127.16.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101}
W20260928 06:33:55.004702 25916 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 06:33:55.086736 25920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:55.087025 25920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:55.087062 25920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:55.088788 25920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:55.088840 25920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:55.088882 25920 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.66
I20260928 06:33:55.091231 25920 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:37789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/krb5kdc/oryx_127.16.157.66.keytab
--principal=oryx/127.16.157.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.66:39763
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.157.66
--webserver_port=39653
--tserver_master_addrs=127.16.157.126:40521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25920
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.66
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:55.091476 25920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:55.091745 25920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:55.097057 25925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:55.097064 25928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:55.097275 25926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:55.097450 25920 server_base.cc:1034] running on GCE node
I20260928 06:33:55.097664 25920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:55.097925 25920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:55.099088 25920 hybrid_clock.cc:648] HybridClock initialized: now 1790577235099075 us; error 28 us; skew 500 ppm
Sep 28 06:33:55 dist-test-slave-rrj5 krb5kdc[25181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577235, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.157.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:55.102401 25920 init.cc:377] Logged in from keytab as oryx/127.16.157.66@KRBTEST.COM (short username oryx)
I20260928 06:33:55.102926 25920 webserver.cc:460] Webserver started at http://127.16.157.66:39653/ using document root <none> and password file <none>
I20260928 06:33:55.103219 25920 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:55.103304 25920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:55.105074 25920 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:33:55.105798 25935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:55.105986 25920 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:55.106061 25920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-1/wal
uuid: "452c125941c846538f27dc86e08e7281"
format_stamp: "Formatted at 2026-09-28 06:33:54 on dist-test-slave-rrj5"
server_key: "6372b410dea5538904b6378db79c0248"
server_key_iv: "192f78624023fc979a2e9f2c0a7f4daa"
server_key_version: "encryptionkey@0"
I20260928 06:33:55.106472 25920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:55.128805 25920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:55.131484 25920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:55.131621 25920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:55.131870 25920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:33:55.132181 25920 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:33:55.132223 25920 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:55.132259 25920 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:33:55.132294 25920 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:55.139318 25920 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.66:39763
I20260928 06:33:55.139395 26051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.66:39763 every 8 connection(s)
I20260928 06:33:55.139721 25920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
Sep 28 06:33:55 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.66@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:33:55 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.66@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 06:33:55.147534 25919 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:33:55.140626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.66:48317 (local address 127.16.157.126:40521)
0928 06:33:55.140672 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:55.140674 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:55.140903 (+ 229us) server_negotiation.cc:407] Connection header received
0928 06:33:55.141474 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:55.141475 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:55.141494 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:55.141556 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:55.142512 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:55.143029 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:55.144175 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:55.144387 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:55.144466 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:55.147479 (+ 3013us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.157.66:48317: BlockingRecv error: recv got EOF from 127.16.157.66:48317 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:33:55.147682 26054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:55.140621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40521 (local address 127.16.157.66:48317)
0928 06:33:55.140772 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:55.140806 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:55.141356 (+ 550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:55.141665 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:55.141669 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:55.142214 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:55.142415 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:55.142421 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:55.143140 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:55.143141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:55.144049 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:55.144052 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:55.144472 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:55.144473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:55.144594 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:55.145114 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:55.145122 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:55.147405 (+ 2283us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.157.126:40521: Server oryx/127.16.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":79,"threads_started":1}
I20260928 06:33:55.148015 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 25920
I20260928 06:33:55.148315 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.67:38307
--local_ip_for_outbound_sockets=127.16.157.67
--tserver_master_addrs=127.16.157.126:40521
--webserver_port=45725
--webserver_interface=127.16.157.67
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab
--principal=kudu/127.16.157.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:37789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.16.157.67
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/krb5kdc/oryx_127.16.157.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:55.149770 26052 heartbeater.cc:644] Failed to heartbeat to 127.16.157.126:40521 (0 consecutive failures): Not authorized: Failed to ping master at 127.16.157.126:40521: Client connection negotiation failed: client connection to 127.16.157.126:40521: Server oryx/127.16.157.126@KRBTEST.COM not found in Kerberos database
Sep 28 06:33:55 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.66@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:33:55 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.66@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:33:55.154817 26056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:55.150010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40521 (local address 127.16.157.66:53347)
0928 06:33:55.150146 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:55.150157 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:55.150208 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:55.150350 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:55.150350 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:55.150507 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:55.150548 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:55.150550 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:55.151204 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:55.151205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:55.151806 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:55.151808 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:55.152172 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:55.152173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:55.152235 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:55.152723 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:55.152727 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:55.154728 (+ 2001us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.157.126:40521: Server oryx/127.16.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":83,"threads_started":1}
W20260928 06:33:55.154886 25919 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:33:55.150210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.66:53347 (local address 127.16.157.126:40521)
0928 06:33:55.150247 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:55.150250 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:55.150256 (+ 6us) server_negotiation.cc:407] Connection header received
0928 06:33:55.150281 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:55.150282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:55.150292 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:55.150347 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:55.150630 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:55.151108 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:55.151874 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:55.152062 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:55.152146 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:55.154816 (+ 2670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.157.66:53347: BlockingRecv error: recv got EOF from 127.16.157.66:53347 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 28 06:33:55 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.66@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:33:55 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.66@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:33:55.162822 26056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:55.155345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40521 (local address 127.16.157.66:46203)
0928 06:33:55.155383 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:55.155391 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:55.155426 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:55.157817 (+ 2391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:55.157819 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:55.157942 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:55.157970 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:55.157971 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:55.158886 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:55.158887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:55.159712 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:55.159714 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:55.160067 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:55.160068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:55.160127 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:55.160551 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:55.160556 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:55.162733 (+ 2177us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.157.126:40521: Server oryx/127.16.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 06:33:55.162878 25919 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:33:55.157634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.66:46203 (local address 127.16.157.126:40521)
0928 06:33:55.157704 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:55.157706 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:55.157715 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:33:55.157737 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:55.157738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:55.157747 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:55.157799 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:55.158054 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:55.158812 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:55.159787 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:55.159994 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:55.160069 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:55.162808 (+ 2739us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.157.66:46203: BlockingRecv error: recv got EOF from 127.16.157.66:46203 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260928 06:33:55.163162 26052 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 06:33:55.248106 26055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:55.248325 26055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:55.248368 26055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:55.249491 26055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:55.249531 26055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:55.249557 26055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.67
I20260928 06:33:55.250982 26055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:37789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/krb5kdc/oryx_127.16.157.67.keytab
--principal=oryx/127.16.157.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.67:38307
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.157.67
--webserver_port=45725
--tserver_master_addrs=127.16.157.126:40521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.67
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:55.251210 26055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:55.251431 26055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:55.256314 26062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:55.256412 26064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:55.256315 26061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:55.256714 26055 server_base.cc:1034] running on GCE node
I20260928 06:33:55.256919 26055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:55.257159 26055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:55.258330 26055 hybrid_clock.cc:648] HybridClock initialized: now 1790577235258309 us; error 31 us; skew 500 ppm
Sep 28 06:33:55 dist-test-slave-rrj5 krb5kdc[25181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577235, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.157.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:55.261431 26055 init.cc:377] Logged in from keytab as oryx/127.16.157.67@KRBTEST.COM (short username oryx)
I20260928 06:33:55.261955 26055 webserver.cc:460] Webserver started at http://127.16.157.67:45725/ using document root <none> and password file <none>
I20260928 06:33:55.262221 26055 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:55.262305 26055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:55.264035 26055 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:33:55.264698 26071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:55.264900 26055 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:33:55.264992 26055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-2/wal
uuid: "1c863190a3ff41f48cbf546ff5486728"
format_stamp: "Formatted at 2026-09-28 06:33:54 on dist-test-slave-rrj5"
server_key: "926e14a0f025c5135797d5d0232db2ca"
server_key_iv: "6ecc269a33ce73bd5dfaa128b686a12d"
server_key_version: "encryptionkey@0"
I20260928 06:33:55.265409 26055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:55.293607 26055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:55.296538 26055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:55.296710 26055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:55.296980 26055 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:33:55.297366 26055 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:33:55.297413 26055 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:55.297454 26055 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:33:55.297483 26055 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:55.305263 26055 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.67:38307
I20260928 06:33:55.305325 26185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.67:38307 every 8 connection(s)
I20260928 06:33:55.305718 26055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
Sep 28 06:33:55 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.67@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:33:55.314247 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 26055
Sep 28 06:33:55 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.67@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 06:33:55.315436 25919 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:33:55.306742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:58069 (local address 127.16.157.126:40521)
0928 06:33:55.306781 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:55.306783 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:55.306795 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:33:55.307566 (+ 771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:55.307568 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:55.307580 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:55.307641 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:55.308971 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:55.309730 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:55.311216 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:55.311550 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:55.311611 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:55.315359 (+ 3748us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.157.67:58069: BlockingRecv error: recv got EOF from 127.16.157.67:58069 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:33:55.315531 26188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:55.306472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40521 (local address 127.16.157.67:58069)
0928 06:33:55.306616 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:55.306694 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:55.307428 (+ 734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:55.307748 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:55.307753 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:55.308562 (+ 809us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:55.308857 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:55.308863 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:55.309806 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:55.309807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:55.311109 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:55.311111 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:55.311687 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:55.311689 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:55.311828 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:55.312392 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:55.312403 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:55.315299 (+ 2896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.157.126:40521: Server oryx/127.16.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":83,"threads_started":1}
W20260928 06:33:55.315814 26186 heartbeater.cc:644] Failed to heartbeat to 127.16.157.126:40521 (0 consecutive failures): Not authorized: Failed to ping master at 127.16.157.126:40521: Client connection negotiation failed: client connection to 127.16.157.126:40521: Server oryx/127.16.157.126@KRBTEST.COM not found in Kerberos database
Sep 28 06:33:55 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.67@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:33:55 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.67@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:33:55.321111 26188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:55.316002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40521 (local address 127.16.157.67:35869)
0928 06:33:55.316056 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:55.316067 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:55.316114 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:55.316223 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:55.316223 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:55.316375 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:55.316423 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:55.316425 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:55.317254 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:55.317254 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:55.318029 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:55.318032 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:55.318501 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:55.318501 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:55.318558 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:55.319066 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:55.319070 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:55.321046 (+ 1976us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.157.126:40521: Server oryx/127.16.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 06:33:55.321199 25919 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:33:55.316089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:35869 (local address 127.16.157.126:40521)
0928 06:33:55.316124 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:55.316126 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:55.316135 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:33:55.316158 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:55.316159 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:55.316166 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:55.316220 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:55.316499 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:55.317161 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:55.318112 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:55.318385 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:55.318426 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:55.321129 (+ 2703us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.157.67:35869: BlockingRecv error: recv got EOF from 127.16.157.67:35869 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 06:33:55.324325 26190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:55.314850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50408 (local address 127.16.157.126:40521)
0928 06:33:55.314982 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:55.314984 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:55.314992 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:33:55.315020 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:55.315021 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:55.315029 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:55.315094 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:55.316292 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:55.316972 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:55.318070 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:55.318385 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:55.318438 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:55.320012 (+ 1574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:55.320017 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:55.320017 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:55.320028 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:55.322302 (+ 2274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:55.323229 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:55.323232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:55.323232 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:55.323354 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:55.323568 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:55.323569 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:55.323570 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:55.323853 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:55.323987 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:55.324119 (+ 132us) server_negotiation.cc:299] Negotiation successful
0928 06:33:55.324241 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":97,"thread_start_us":72,"threads_started":1}
Sep 28 06:33:55 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.67@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:33:55 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.67@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:33:55.326671 26188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:55.321595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40521 (local address 127.16.157.67:50685)
0928 06:33:55.321692 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:55.321702 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:55.321750 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:55.321861 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:55.321861 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:55.321993 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:55.322021 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:55.322022 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:55.322893 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:55.322894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:55.323697 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:55.323699 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:55.324299 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:55.324300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:55.324356 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:55.324857 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:55.324861 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:55.326601 (+ 1740us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.157.126:40521: Server oryx/127.16.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":83}
W20260928 06:33:55.326722 25919 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:33:55.321648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:50685 (local address 127.16.157.126:40521)
0928 06:33:55.321703 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:55.321705 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:55.321738 (+ 33us) server_negotiation.cc:407] Connection header received
0928 06:33:55.321796 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:55.321796 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:55.321804 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:55.321860 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:55.322110 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:55.322820 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:55.323757 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:55.324020 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:55.324077 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:55.326661 (+ 2584us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.157.67:50685: BlockingRecv error: recv got EOF from 127.16.157.67:50685 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260928 06:33:55.326900 26186 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 06:33:56 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.65@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:33:56 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.65@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:33:56.010505 26191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:56.005093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40521 (local address 127.16.157.65:46359)
0928 06:33:56.005225 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:56.005237 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:56.005295 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:56.005476 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:56.005477 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:56.005696 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:56.005747 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:56.005749 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:56.006440 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:56.006441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:56.007018 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:56.007020 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:56.007406 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:56.007407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:56.007466 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:56.008105 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:56.008134 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:56.010392 (+ 2258us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.157.126:40521: Server oryx/127.16.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":82,"threads_started":1}
W20260928 06:33:56.010582 26192 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:33:56.005161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:46359 (local address 127.16.157.126:40521)
0928 06:33:56.005287 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:56.005289 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:56.005300 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:33:56.005383 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:56.005384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:56.005396 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:56.005470 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:56.005838 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:56.006348 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:56.007108 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:56.007312 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:56.007403 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:56.010509 (+ 3106us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.157.65:46359: BlockingRecv error: recv got EOF from 127.16.157.65:46359 (error 108)
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":69,"threads_started":1}
Sep 28 06:33:56 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.66@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:33:56 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.66@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:33:56.171264 26193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:56.163552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40521 (local address 127.16.157.66:40017)
0928 06:33:56.163664 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:56.163676 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:56.163736 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:56.163895 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:56.163896 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:56.164126 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:56.164191 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:56.164194 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:56.165176 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:56.165177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:56.165912 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:56.165915 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:56.166355 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:56.166356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:56.166417 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:56.166894 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:56.166899 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:56.171140 (+ 4241us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.157.126:40521: Server oryx/127.16.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":65,"threads_started":1}
W20260928 06:33:56.171319 26192 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:33:56.163721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.66:40017 (local address 127.16.157.126:40521)
0928 06:33:56.163774 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:56.163776 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:56.163786 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:33:56.163812 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:56.163813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:56.163824 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:56.163882 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:56.164262 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:56.165076 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:56.166055 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:56.166259 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:56.166334 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:56.171245 (+ 4911us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.157.66:40017: BlockingRecv error: recv got EOF from 127.16.157.66:40017 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 28 06:33:56 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.67@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:33:56 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.67@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:33:56.333267 26194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:56.327290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40521 (local address 127.16.157.67:40151)
0928 06:33:56.327411 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:56.327421 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:56.327485 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:56.327671 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:56.327672 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:56.327906 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:56.327992 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:56.327995 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:56.328968 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:56.328969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:56.329892 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:56.329894 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:56.330392 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:56.330393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:56.330454 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:56.331012 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:56.331017 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:56.333199 (+ 2182us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.157.126:40521: Server oryx/127.16.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":65,"threads_started":1}
W20260928 06:33:56.333417 26192 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:33:56.327333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:40151 (local address 127.16.157.126:40521)
0928 06:33:56.327405 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:56.327408 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:56.327474 (+ 66us) server_negotiation.cc:407] Connection header received
0928 06:33:56.327543 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:56.327543 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:56.327559 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:56.327637 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:56.328077 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:56.328883 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:56.329980 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:56.330305 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:56.330391 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:56.333304 (+ 2913us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.157.67:40151: BlockingRecv error: recv got EOF from 127.16.157.67:40151 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 28 06:33:57 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.65@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:33:57 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.65@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:33:57.017076 26195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:57.010982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40521 (local address 127.16.157.65:56325)
0928 06:33:57.011139 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:57.011152 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:57.011235 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:57.011415 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:57.011435 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:57.011661 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:57.011726 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:57.011729 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:57.012740 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:57.012742 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:57.013781 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:57.013785 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:57.014295 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:57.014297 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:57.014386 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:57.014955 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:57.014961 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:57.017016 (+ 2055us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.157.126:40521: Server oryx/127.16.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":82,"threads_started":1}
W20260928 06:33:57.017189 26196 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:33:57.011119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:56325 (local address 127.16.157.126:40521)
0928 06:33:57.011221 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:57.011224 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:57.011233 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:33:57.011313 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:57.011313 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:57.011328 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:57.011399 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:57.011811 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:57.012654 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:57.013889 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:57.014216 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:57.014282 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:57.017113 (+ 2831us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.157.65:56325: BlockingRecv error: recv got EOF from 127.16.157.65:56325 (error 108)
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":49,"threads_started":1}
Sep 28 06:33:57 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.66@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:33:57 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.66@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:33:57.177227 26197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:57.171969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40521 (local address 127.16.157.66:36405)
0928 06:33:57.172146 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:57.172163 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:57.172238 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:57.172401 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:57.172401 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:57.172579 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:57.172627 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:57.172628 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:57.173316 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:57.173318 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:57.174151 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:57.174154 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:57.174592 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:57.174593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:57.174652 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:57.175194 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:57.175198 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:57.177115 (+ 1917us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.157.126:40521: Server oryx/127.16.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":109,"threads_started":1}
W20260928 06:33:57.177340 26196 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:33:57.172035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.66:36405 (local address 127.16.157.126:40521)
0928 06:33:57.172108 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:57.172110 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:57.172228 (+ 118us) server_negotiation.cc:407] Connection header received
0928 06:33:57.172303 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:57.172304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:57.172316 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:57.172393 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:57.172713 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:57.173224 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:57.174246 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:57.174506 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:57.174591 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:57.177260 (+ 2669us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.157.66:36405: BlockingRecv error: recv got EOF from 127.16.157.66:36405 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 28 06:33:57 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.67@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:33:57 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.67@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:33:57.339983 26198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:57.333778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40521 (local address 127.16.157.67:50429)
0928 06:33:57.333907 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:57.333919 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:57.333979 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:57.334289 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:57.334290 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:57.334509 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:57.334582 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:57.334584 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:57.335655 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:57.335657 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:57.336528 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:57.336530 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:57.337063 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:57.337064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:57.337132 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:57.337704 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:57.337709 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:57.339924 (+ 2215us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.157.126:40521: Server oryx/127.16.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":79,"threads_started":1}
W20260928 06:33:57.340145 26196 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:33:57.333903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:50429 (local address 127.16.157.126:40521)
0928 06:33:57.334057 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:57.334060 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:57.334148 (+ 88us) server_negotiation.cc:407] Connection header received
0928 06:33:57.334177 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:57.334178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:57.334192 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:57.334253 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:57.334665 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:57.335544 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:57.336625 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:57.336958 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:57.337065 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:57.340058 (+ 2993us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.157.67:50429: BlockingRecv error: recv got EOF from 127.16.157.67:50429 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 28 06:33:58 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.65@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:33:58 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.65@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:33:58.023162 26199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:58.017558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40521 (local address 127.16.157.65:52923)
0928 06:33:58.017712 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:58.017727 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:58.017787 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:58.018146 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:58.018148 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:58.018372 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:58.018422 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:58.018424 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:58.019219 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:58.019220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:58.019752 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:58.019753 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:58.020147 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:58.020148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:58.020236 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:58.020798 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:58.020805 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:58.023071 (+ 2266us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.157.126:40521: Server oryx/127.16.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":100,"threads_started":1}
W20260928 06:33:58.023279 26200 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:33:58.017799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:52923 (local address 127.16.157.126:40521)
0928 06:33:58.017958 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:58.017960 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:58.017969 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:33:58.017997 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:58.017998 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:58.018010 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:58.018076 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:58.018545 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:58.019103 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:58.019871 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:58.020064 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:58.020115 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:58.023209 (+ 3094us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.157.65:52923: BlockingRecv error: recv got EOF from 127.16.157.65:52923 (error 108)
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":79,"threads_started":1}
Sep 28 06:33:58 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.66@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:33:58 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.66@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:33:58.184262 26201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:58.177767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40521 (local address 127.16.157.66:54887)
0928 06:33:58.177937 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:58.178014 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:58.178074 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:58.178386 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:58.178387 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:58.178603 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:58.178682 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:58.178684 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:58.179857 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:58.179858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:58.180703 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:58.180706 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:58.181314 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:58.181315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:58.181377 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:58.181988 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:58.181994 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:58.184176 (+ 2182us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.157.126:40521: Server oryx/127.16.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":101,"threads_started":1}
W20260928 06:33:58.184376 26200 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:33:58.178153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.66:54887 (local address 127.16.157.126:40521)
0928 06:33:58.178226 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:58.178229 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:58.178241 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:33:58.178275 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:58.178276 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:58.178293 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:58.178379 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:58.178780 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:58.179744 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:58.180816 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:58.181222 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:58.181321 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:58.184314 (+ 2993us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.157.66:54887: BlockingRecv error: recv got EOF from 127.16.157.66:54887 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 28 06:33:58 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.67@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:33:58 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.67@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:33:58.345548 26202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:58.340665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40521 (local address 127.16.157.67:54653)
0928 06:33:58.340826 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:58.340836 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:58.340912 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:58.341060 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:58.341061 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:58.341277 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:58.341326 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:58.341328 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:58.342110 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:58.342111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:58.342671 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:58.342673 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:58.343079 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:58.343080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:58.343144 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:58.343652 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:58.343658 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:58.345450 (+ 1792us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.157.126:40521: Server oryx/127.16.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":88,"threads_started":1}
W20260928 06:33:58.345633 26200 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:33:58.340717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:54653 (local address 127.16.157.126:40521)
0928 06:33:58.340781 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:58.340783 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:58.340895 (+ 112us) server_negotiation.cc:407] Connection header received
0928 06:33:58.340973 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:58.340974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:58.340988 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:58.341045 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:58.341404 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:58.342022 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:58.342779 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:58.342992 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:58.343076 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:58.345530 (+ 2454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.157.67:54653: BlockingRecv error: recv got EOF from 127.16.157.67:54653 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 28 06:33:59 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.65@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:33:59 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.65@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:33:59.029235 26203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:59.023831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40521 (local address 127.16.157.65:45103)
0928 06:33:59.023966 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:59.023983 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:59.024031 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:59.024223 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:59.024225 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:59.024417 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:59.024484 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:59.024487 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:59.025173 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:59.025175 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:59.025956 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:59.025959 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:59.026363 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:59.026364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:59.026421 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:59.026938 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:59.026943 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:59.029147 (+ 2204us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.157.126:40521: Server oryx/127.16.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":84,"threads_started":1}
W20260928 06:33:59.029348 26204 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:33:59.023893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:45103 (local address 127.16.157.126:40521)
0928 06:33:59.024010 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:59.024012 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:59.024023 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:33:59.024126 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:59.024127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:59.024139 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:59.024201 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:59.024558 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:59.025070 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:59.026056 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:59.026273 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:59.026360 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:59.029250 (+ 2890us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.157.65:45103: BlockingRecv error: recv got EOF from 127.16.157.65:45103 (error 108)
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":58,"threads_started":1}
Sep 28 06:33:59 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.66@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:33:59 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.66@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:33:59.190436 26205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:59.184840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40521 (local address 127.16.157.66:58009)
0928 06:33:59.184993 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:59.185003 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:59.185103 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:59.185257 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:59.185258 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:59.185483 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:59.185551 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:59.185553 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:59.186538 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:59.186539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:59.187310 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:59.187312 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:59.187705 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:59.187706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:59.187782 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:59.188271 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:59.188276 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:59.190327 (+ 2051us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.157.126:40521: Server oryx/127.16.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":83,"threads_started":1}
W20260928 06:33:59.190689 26204 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:33:59.184884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.66:58009 (local address 127.16.157.126:40521)
0928 06:33:59.184962 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:59.184965 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:59.185083 (+ 118us) server_negotiation.cc:407] Connection header received
0928 06:33:59.185163 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:59.185164 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:59.185176 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:59.185252 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:59.185653 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:59.186458 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:59.187421 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:59.187632 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:59.187708 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:59.190436 (+ 2728us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.157.66:58009: BlockingRecv error: recv got EOF from 127.16.157.66:58009 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 28 06:33:59 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.67@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:33:59 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.67@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:33:59.351753 26206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:59.346020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40521 (local address 127.16.157.67:58849)
0928 06:33:59.346150 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:59.346161 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:59.346233 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:59.346432 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:59.346433 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:59.346667 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:59.346731 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:59.346734 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:59.347694 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:59.347695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:59.348556 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:59.348559 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:59.348948 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:59.348949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:59.349072 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:59.349560 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:59.349565 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:59.351671 (+ 2106us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.157.126:40521: Server oryx/127.16.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":66,"threads_started":1}
W20260928 06:33:59.351910 26204 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:33:59.346099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:58849 (local address 127.16.157.126:40521)
0928 06:33:59.346150 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:59.346152 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:59.346235 (+ 83us) server_negotiation.cc:407] Connection header received
0928 06:33:59.346312 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:59.346331 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:59.346345 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:59.346407 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:59.346807 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:59.347608 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:59.348644 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:59.348860 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:59.348948 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:59.351799 (+ 2851us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.157.67:58849: BlockingRecv error: recv got EOF from 127.16.157.67:58849 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 28 06:34:00 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.65@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:34:00 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.65@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:34:00.039834 26207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:00.029883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40521 (local address 127.16.157.65:41847)
0928 06:34:00.030073 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:00.030087 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:00.030161 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:00.030324 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:00.030325 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:00.030555 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:34:00.030629 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:00.030632 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:00.031622 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:00.031623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:00.032546 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:00.032550 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:00.033057 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:00.033058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:00.033155 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:00.037352 (+ 4197us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:34:00.037358 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:34:00.039747 (+ 2389us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.157.126:40521: Server oryx/127.16.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":64,"threads_started":1}
W20260928 06:34:00.040004 26208 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:34:00.029934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:41847 (local address 127.16.157.126:40521)
0928 06:34:00.030055 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:00.030058 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:00.030158 (+ 100us) server_negotiation.cc:407] Connection header received
0928 06:34:00.030232 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:00.030233 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:00.030245 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:00.030305 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:00.030718 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:00.031528 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:00.032650 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:00.032981 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:00.033051 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:00.039941 (+ 6890us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.157.65:41847: BlockingRecv error: recv got EOF from 127.16.157.65:41847 (error 108)
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":68,"threads_started":1}
Sep 28 06:34:00 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.66@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:34:00 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.66@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 06:34:00.197389 26208 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:34:00.190980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.66:43221 (local address 127.16.157.126:40521)
0928 06:34:00.191034 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:00.191036 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:00.191174 (+ 138us) server_negotiation.cc:407] Connection header received
0928 06:34:00.191256 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:00.191257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:00.191268 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:00.191339 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:00.191859 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:00.192691 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:00.193806 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:00.194128 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:00.194198 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:00.197330 (+ 3132us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.157.66:43221: BlockingRecv error: recv got EOF from 127.16.157.66:43221 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 06:34:00.197458 26209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:00.190893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40521 (local address 127.16.157.66:43221)
0928 06:34:00.191092 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:00.191105 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:00.191194 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:00.191415 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:00.191416 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:00.191675 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:34:00.191736 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:00.191738 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:00.192800 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:00.192801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:00.193733 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:00.193736 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:00.194207 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:00.194208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:00.194267 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:00.194776 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:34:00.194781 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:34:00.197238 (+ 2457us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.157.126:40521: Server oryx/127.16.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":66,"threads_started":1}
Sep 28 06:34:00 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.67@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:34:00 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.67@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:34:00.358395 26210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:00.352234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40521 (local address 127.16.157.67:39789)
0928 06:34:00.352373 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:00.352384 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:00.352454 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:00.352679 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:00.352680 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:00.352910 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:34:00.352979 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:00.352981 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:00.353987 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:00.353989 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:00.354964 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:00.354967 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:00.355448 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:00.355449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:00.355495 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:00.356031 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:34:00.356036 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:34:00.358313 (+ 2277us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.157.126:40521: Server oryx/127.16.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":72,"threads_started":1}
W20260928 06:34:00.358495 26208 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:34:00.352288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:39789 (local address 127.16.157.126:40521)
0928 06:34:00.352338 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:00.352340 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:00.352471 (+ 131us) server_negotiation.cc:407] Connection header received
0928 06:34:00.352505 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:00.352506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:00.352520 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:00.352621 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:00.353059 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:00.353896 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:00.355063 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:00.355375 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:00.355440 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:00.358417 (+ 2977us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.157.67:39789: BlockingRecv error: recv got EOF from 127.16.157.67:39789 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 28 06:34:01 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.65@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:34:01 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.65@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:34:01.046638 26211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:01.040550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40521 (local address 127.16.157.65:43593)
0928 06:34:01.040747 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:01.040757 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:01.040813 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:01.040980 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:01.040981 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:01.041175 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:34:01.041230 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:01.041231 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:01.042331 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:01.042332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:01.042942 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:01.042944 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:01.043500 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:01.043501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:01.043571 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:01.044153 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:34:01.044158 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:34:01.046536 (+ 2378us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.157.126:40521: Server oryx/127.16.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":78,"threads_started":1}
W20260928 06:34:01.046947 26212 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:34:01.040630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:43593 (local address 127.16.157.126:40521)
0928 06:34:01.040754 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:01.040757 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:01.040800 (+ 43us) server_negotiation.cc:407] Connection header received
0928 06:34:01.040879 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:01.040880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:01.040894 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:01.040970 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:01.041290 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:01.042245 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:01.043045 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:01.043406 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:01.043489 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:01.046670 (+ 3181us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.157.65:43593: BlockingRecv error: recv got EOF from 127.16.157.65:43593 (error 108)
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":51,"threads_started":1}
Sep 28 06:34:01 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.66@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:34:01 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.66@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:34:01.203644 26213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:01.197970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40521 (local address 127.16.157.66:37409)
0928 06:34:01.198119 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:01.198129 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:01.198193 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:01.198363 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:01.198364 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:01.198584 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:34:01.198646 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:01.198649 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:01.199618 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:01.199620 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:01.200503 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:01.200507 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:01.200995 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:01.200996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:01.201072 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:01.201755 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:34:01.201761 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:34:01.203583 (+ 1822us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.157.126:40521: Server oryx/127.16.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":81,"threads_started":1}
W20260928 06:34:01.203807 26212 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:34:01.198048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.66:37409 (local address 127.16.157.126:40521)
0928 06:34:01.198117 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:01.198120 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:01.198175 (+ 55us) server_negotiation.cc:407] Connection header received
0928 06:34:01.198266 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:01.198267 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:01.198282 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:01.198348 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:01.198722 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:01.199526 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:01.200599 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:01.200927 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:01.200997 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:01.203697 (+ 2700us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.157.66:37409: BlockingRecv error: recv got EOF from 127.16.157.66:37409 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 28 06:34:01 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.67@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:34:01 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.67@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 06:34:01.364384 26212 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:34:01.358986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:38113 (local address 127.16.157.126:40521)
0928 06:34:01.359040 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:01.359042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:01.359118 (+ 76us) server_negotiation.cc:407] Connection header received
0928 06:34:01.359170 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:01.359171 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:01.359186 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:01.359242 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:01.359532 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:01.360342 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:01.361040 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:01.361383 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:01.361457 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:01.364333 (+ 2876us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.157.67:38113: BlockingRecv error: recv got EOF from 127.16.157.67:38113 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 06:34:01.364367 26214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:01.358906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40521 (local address 127.16.157.67:38113)
0928 06:34:01.359042 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:01.359052 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:01.359111 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:01.359254 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:01.359254 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:01.359404 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:34:01.359454 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:01.359455 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:01.360415 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:01.360415 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:01.360954 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:01.360956 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:01.361461 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:01.361462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:01.361503 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:01.362048 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:34:01.362055 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:34:01.364263 (+ 2208us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.157.126:40521: Server oryx/127.16.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":67,"threads_started":1}
Sep 28 06:34:02 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.65@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:34:02 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.65@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:34:02.053045 26215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:02.047123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40521 (local address 127.16.157.65:44475)
0928 06:34:02.047287 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:02.047300 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:02.047359 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:02.047605 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:02.047607 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:02.047860 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:34:02.047923 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:02.047925 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:02.049063 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:02.049065 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:02.049939 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:02.049942 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:02.050583 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:02.050584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:02.050648 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:02.051190 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:34:02.051194 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:34:02.052985 (+ 1791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.157.126:40521: Server oryx/127.16.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":72,"threads_started":1}
W20260928 06:34:02.053220 26216 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:34:02.047256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:44475 (local address 127.16.157.126:40521)
0928 06:34:02.047405 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:02.047407 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:02.047420 (+ 13us) server_negotiation.cc:407] Connection header received
0928 06:34:02.047446 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:02.047447 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:02.047462 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:02.047566 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:02.048028 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:02.048964 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:02.050115 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:02.050450 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:02.050518 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:02.053118 (+ 2600us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.157.65:44475: BlockingRecv error: recv got EOF from 127.16.157.65:44475 (error 108)
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":77,"threads_started":1}
Sep 28 06:34:02 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.66@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:34:02 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.66@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:34:02.208958 26217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:02.204112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40521 (local address 127.16.157.66:47697)
0928 06:34:02.204253 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:02.204264 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:02.204352 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:02.204502 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:02.204503 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:02.204684 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:34:02.204737 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:02.204739 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:02.205419 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:02.205420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:02.206057 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:02.206060 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:02.206439 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:02.206440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:02.206499 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:02.206988 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:34:02.206993 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:34:02.208863 (+ 1870us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.157.126:40521: Server oryx/127.16.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":80,"threads_started":1}
W20260928 06:34:02.209007 26216 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:34:02.204153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.66:47697 (local address 127.16.157.126:40521)
0928 06:34:02.204220 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:02.204223 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:02.204343 (+ 120us) server_negotiation.cc:407] Connection header received
0928 06:34:02.204408 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:02.204408 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:02.204422 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:02.204496 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:02.204821 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:02.205335 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:02.206152 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:02.206355 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:02.206429 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:02.208943 (+ 2514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.157.66:47697: BlockingRecv error: recv got EOF from 127.16.157.66:47697 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 28 06:34:02 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.67@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:34:02 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.67@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:34:02.369915 26218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:02.364836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40521 (local address 127.16.157.67:36265)
0928 06:34:02.364979 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:02.364989 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:02.365067 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:02.365216 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:02.365217 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:02.365417 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:34:02.365464 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:02.365466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:02.366259 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:02.366260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:02.366824 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:02.366825 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:02.367175 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:02.367176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:02.367231 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:02.367717 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:34:02.367722 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:34:02.369821 (+ 2099us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.157.126:40521: Server oryx/127.16.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":76,"threads_started":1}
W20260928 06:34:02.369956 26216 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:34:02.364887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:36265 (local address 127.16.157.126:40521)
0928 06:34:02.364941 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:02.364943 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:02.365054 (+ 111us) server_negotiation.cc:407] Connection header received
0928 06:34:02.365127 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:02.365127 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:02.365140 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:02.365206 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:02.365539 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:02.366181 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:02.366901 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:02.367103 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:02.367183 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:02.369894 (+ 2711us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.157.67:36265: BlockingRecv error: recv got EOF from 127.16.157.67:36265 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 28 06:34:03 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.65@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:34:03 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.65@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:34:03.059208 26219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:03.053489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40521 (local address 127.16.157.65:58317)
0928 06:34:03.053643 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:03.053658 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:03.053724 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:03.054012 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:03.054013 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:03.054253 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:34:03.054317 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:03.054320 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:03.055027 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:03.055028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:03.055719 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:03.055737 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:03.056119 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:03.056120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:03.056178 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:03.056786 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:34:03.056790 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:34:03.059099 (+ 2309us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.157.126:40521: Server oryx/127.16.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":99,"threads_started":1}
W20260928 06:34:03.059295 26220 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:34:03.053736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:58317 (local address 127.16.157.126:40521)
0928 06:34:03.053865 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:03.053867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:03.053890 (+ 23us) server_negotiation.cc:407] Connection header received
0928 06:34:03.053916 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:03.053917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:03.053928 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:03.053995 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:03.054402 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:03.054936 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:03.055815 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:03.056043 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:03.056120 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:03.059194 (+ 3074us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.157.65:58317: BlockingRecv error: recv got EOF from 127.16.157.65:58317 (error 108)
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":77,"threads_started":1}
Sep 28 06:34:03 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.66@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:34:03 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.66@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 06:34:03.215969 26220 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:34:03.209605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.66:47807 (local address 127.16.157.126:40521)
0928 06:34:03.209668 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:03.209670 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:03.209742 (+ 72us) server_negotiation.cc:407] Connection header received
0928 06:34:03.209868 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:03.209869 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:03.209883 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:03.209960 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:03.210422 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:03.211691 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:03.212417 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:03.212764 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:03.212849 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:03.215922 (+ 3073us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.157.66:47807: BlockingRecv error: recv got EOF from 127.16.157.66:47807 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 06:34:03.215970 26221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:03.209519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40521 (local address 127.16.157.66:47807)
0928 06:34:03.209674 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:03.209686 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:03.209790 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:03.209959 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:03.209959 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:03.210245 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:34:03.210331 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:03.210333 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:03.211777 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:03.211778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:03.212331 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:03.212333 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:03.212845 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:03.212846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:03.212889 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:03.213439 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:34:03.213444 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:34:03.215857 (+ 2413us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.157.126:40521: Server oryx/127.16.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":91,"threads_started":1}
Sep 28 06:34:03 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.67@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:34:03 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.67@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:34:03.376466 26222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:03.370512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40521 (local address 127.16.157.67:58925)
0928 06:34:03.370731 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:03.370741 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:03.370804 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:03.371023 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:03.371024 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:03.371219 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:34:03.371311 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:03.371314 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:03.372346 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:03.372347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:03.373229 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:03.373231 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:03.373745 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:03.373746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:03.373810 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:03.374400 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:34:03.374405 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:34:03.376355 (+ 1950us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.157.126:40521: Server oryx/127.16.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":109,"threads_started":1}
W20260928 06:34:03.376612 26220 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:34:03.370582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:58925 (local address 127.16.157.126:40521)
0928 06:34:03.370661 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:03.370663 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:03.370815 (+ 152us) server_negotiation.cc:407] Connection header received
0928 06:34:03.370858 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:03.370859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:03.370919 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:03.370997 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:03.371411 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:03.372259 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:03.373321 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:03.373660 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:03.373745 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:03.376502 (+ 2757us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.157.67:58925: BlockingRecv error: recv got EOF from 127.16.157.67:58925 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 28 06:34:04 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.65@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:34:04 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.65@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:34:04.065186 26223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:04.059901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40521 (local address 127.16.157.65:34423)
0928 06:34:04.060045 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:04.060056 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:04.060147 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:04.060354 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:04.060355 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:04.060541 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:34:04.060590 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:04.060592 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:04.061537 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:04.061538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:04.062205 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:04.062207 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:04.062578 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:04.062579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:04.062638 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:04.063178 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:34:04.063184 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:34:04.065086 (+ 1902us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.157.126:40521: Server oryx/127.16.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":69,"threads_started":1}
W20260928 06:34:04.065251 26224 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:34:04.059945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:34423 (local address 127.16.157.126:40521)
0928 06:34:04.060062 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:04.060065 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:04.060129 (+ 64us) server_negotiation.cc:407] Connection header received
0928 06:34:04.060226 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:04.060227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:04.060240 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:04.060347 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:04.060678 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:04.061376 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:04.062301 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:04.062504 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:04.062579 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:04.065176 (+ 2597us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.157.65:34423: BlockingRecv error: recv got EOF from 127.16.157.65:34423 (error 108)
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":47,"threads_started":1}
Sep 28 06:34:04 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.66@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:34:04 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.66@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:34:04.222534 26225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:04.216624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40521 (local address 127.16.157.66:41625)
0928 06:34:04.216763 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:04.216774 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:04.216876 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:04.217009 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:04.217010 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:04.217212 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:34:04.217273 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:04.217275 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:04.218211 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:04.218213 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:04.218864 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:04.218867 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:04.219337 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:04.219338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:04.219401 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:04.219974 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:34:04.219979 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:34:04.222420 (+ 2441us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.157.126:40521: Server oryx/127.16.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":78,"threads_started":1}
W20260928 06:34:04.222582 26224 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:34:04.216665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.66:41625 (local address 127.16.157.126:40521)
0928 06:34:04.216718 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:04.216721 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:04.216850 (+ 129us) server_negotiation.cc:407] Connection header received
0928 06:34:04.216937 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:04.216938 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:04.216951 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:04.217004 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:04.217366 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:04.218111 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:04.218953 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:04.219249 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:04.219326 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:04.222512 (+ 3186us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.157.66:41625: BlockingRecv error: recv got EOF from 127.16.157.66:41625 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 28 06:34:04 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.67@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:34:04 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.67@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:34:04.383914 26226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:04.377174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40521 (local address 127.16.157.67:51261)
0928 06:34:04.377349 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:04.377365 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:04.377446 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:04.377655 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:04.377657 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:04.377904 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:34:04.377978 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:04.377981 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:04.379041 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:04.379042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:04.380010 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:04.380013 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:04.380560 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:04.380561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:04.380636 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:04.381219 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:34:04.381225 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:34:04.383829 (+ 2604us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.157.126:40521: Server oryx/127.16.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":65,"threads_started":1}
W20260928 06:34:04.384042 26224 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:34:04.377329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:51261 (local address 127.16.157.126:40521)
0928 06:34:04.377376 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:04.377378 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:04.377452 (+ 74us) server_negotiation.cc:407] Connection header received
0928 06:34:04.377543 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:04.377544 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:04.377586 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:04.377642 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:04.378078 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:04.378934 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:04.380130 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:04.380456 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:04.380528 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:04.383952 (+ 3424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.157.67:51261: BlockingRecv error: recv got EOF from 127.16.157.67:51261 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 28 06:34:05 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.65@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:34:05 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.65@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:34:05.076323 26227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:05.065943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40521 (local address 127.16.157.65:57371)
0928 06:34:05.066140 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:05.066153 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:05.066225 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:05.066377 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:05.066377 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:05.066588 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:34:05.066661 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:05.066663 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:05.067617 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:05.067619 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:05.068553 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:05.068556 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:05.069052 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:05.069054 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:05.069116 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:05.073664 (+ 4548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:34:05.073671 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:34:05.076223 (+ 2552us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.157.126:40521: Server oryx/127.16.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":77,"threads_started":1}
W20260928 06:34:05.076392 26228 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:34:05.065987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:57371 (local address 127.16.157.126:40521)
0928 06:34:05.066143 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:05.066145 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:05.066205 (+ 60us) server_negotiation.cc:407] Connection header received
0928 06:34:05.066295 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:05.066295 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:05.066308 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:05.066370 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:05.066738 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:05.067517 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:05.068647 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:05.068969 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:05.069035 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:05.076316 (+ 7281us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.157.65:57371: BlockingRecv error: recv got EOF from 127.16.157.65:57371 (error 108)
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":81,"threads_started":1}
Sep 28 06:34:05 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.66@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:34:05 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.66@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:34:05.228999 26229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:05.223225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40521 (local address 127.16.157.66:59287)
0928 06:34:05.223400 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:05.223411 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:05.223522 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:05.223708 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:05.223709 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:05.223950 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:34:05.224029 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:05.224031 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:05.225226 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:05.225227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:05.225862 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:05.225864 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:05.226463 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:05.226463 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:05.226502 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:05.227026 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:34:05.227031 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:34:05.228937 (+ 1906us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.157.126:40521: Server oryx/127.16.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":103,"threads_started":1}
W20260928 06:34:05.229197 26228 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:34:05.223317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.66:59287 (local address 127.16.157.126:40521)
0928 06:34:05.223369 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:05.223371 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:05.223505 (+ 134us) server_negotiation.cc:407] Connection header received
0928 06:34:05.223594 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:05.223596 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:05.223610 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:05.223692 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:05.224118 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:05.225098 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:05.225927 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:05.226368 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:05.226459 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:05.229094 (+ 2635us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.157.66:59287: BlockingRecv error: recv got EOF from 127.16.157.66:59287 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 28 06:34:05 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.67@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:34:05 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.67@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:34:05.390548 26230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:05.384431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40521 (local address 127.16.157.67:40271)
0928 06:34:05.384648 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:05.384658 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:05.384763 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:05.384958 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:05.384959 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:05.385178 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:34:05.385246 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:05.385247 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:05.386285 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:05.386287 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:05.386920 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:05.386922 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:05.387421 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:05.387422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:05.387501 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:05.388061 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:34:05.388066 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:34:05.390428 (+ 2362us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.157.126:40521: Server oryx/127.16.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":81,"threads_started":1}
W20260928 06:34:05.390582 26228 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:34:05.384537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:40271 (local address 127.16.157.126:40521)
0928 06:34:05.384589 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:05.384591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:05.384738 (+ 147us) server_negotiation.cc:407] Connection header received
0928 06:34:05.384863 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:05.384864 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:05.384880 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:05.384928 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:05.385352 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:05.386168 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:05.387042 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:05.387343 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:05.387417 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:05.390504 (+ 3087us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.157.67:40271: BlockingRecv error: recv got EOF from 127.16.157.67:40271 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 28 06:34:06 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.65@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:34:06 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.65@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:34:06.083106 26231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:06.077046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40521 (local address 127.16.157.65:43261)
0928 06:34:06.077185 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:06.077196 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:06.077356 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:06.077569 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:06.077570 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:06.077779 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:34:06.077829 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:06.077831 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:06.079001 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:06.079002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:06.079612 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:06.079615 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:06.080226 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:06.080227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:06.080288 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:06.080818 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:34:06.080822 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:34:06.083020 (+ 2198us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.157.126:40521: Server oryx/127.16.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":62,"threads_started":1}
W20260928 06:34:06.083251 26232 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:34:06.077114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:43261 (local address 127.16.157.126:40521)
0928 06:34:06.077232 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:06.077235 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:06.077249 (+ 14us) server_negotiation.cc:407] Connection header received
0928 06:34:06.077435 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:06.077436 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:06.077453 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:06.077548 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:06.077915 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:06.078911 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:06.079703 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:06.080116 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:06.080201 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:06.083146 (+ 2945us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.157.65:43261: BlockingRecv error: recv got EOF from 127.16.157.65:43261 (error 108)
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":51,"threads_started":1}
Sep 28 06:34:06 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.66@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:34:06 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.66@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 06:34:06.236123 26232 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:34:06.229558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.66:51713 (local address 127.16.157.126:40521)
0928 06:34:06.229679 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:06.229682 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:06.229753 (+ 71us) server_negotiation.cc:407] Connection header received
0928 06:34:06.229814 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:06.229815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:06.229831 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:06.229924 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:06.230257 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:06.231363 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:06.232276 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:06.232682 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:06.232770 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:06.236074 (+ 3304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.157.66:51713: BlockingRecv error: recv got EOF from 127.16.157.66:51713 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 06:34:06.236117 26233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:06.229515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40521 (local address 127.16.157.66:51713)
0928 06:34:06.229665 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:06.229676 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:06.229745 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:06.229924 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:06.229925 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:06.230119 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:34:06.230171 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:06.230173 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:06.231492 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:06.231493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:06.232127 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:06.232130 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:06.232769 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:06.232770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:06.232821 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:06.233456 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:34:06.233468 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:34:06.235984 (+ 2516us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.157.126:40521: Server oryx/127.16.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":102,"threads_started":1}
Sep 28 06:34:06 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.67@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:34:06 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.67@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:34:06.396492 26234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:06.391036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40521 (local address 127.16.157.67:34975)
0928 06:34:06.391203 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:06.391224 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:06.391317 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:06.391486 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:06.391487 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:06.391714 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:34:06.391761 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:06.391763 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:06.392446 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:06.392448 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:06.393027 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:06.393029 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:06.393403 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:06.393404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:06.393466 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:06.394006 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:34:06.394029 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:34:06.396388 (+ 2359us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.157.126:40521: Server oryx/127.16.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":105,"threads_started":1}
W20260928 06:34:06.396528 26232 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:34:06.391139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:34975 (local address 127.16.157.126:40521)
0928 06:34:06.391207 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:06.391210 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:06.391321 (+ 111us) server_negotiation.cc:407] Connection header received
0928 06:34:06.391387 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:06.391388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:06.391402 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:06.391479 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:06.391848 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:06.392355 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:06.393121 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:06.393324 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:06.393403 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:06.396466 (+ 3063us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.157.67:34975: BlockingRecv error: recv got EOF from 127.16.157.67:34975 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 28 06:34:07 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.65@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:34:07 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.65@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:34:07.088970 26235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:07.083642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40521 (local address 127.16.157.65:47295)
0928 06:34:07.083781 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:07.083794 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:07.083863 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:07.084044 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:07.084045 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:07.084228 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:34:07.084275 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:07.084277 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:07.084990 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:07.084991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:07.085537 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:07.085538 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:07.086068 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:07.086069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:07.086141 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:07.086631 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:34:07.086639 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:34:07.088872 (+ 2233us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.157.126:40521: Server oryx/127.16.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":72,"threads_started":1}
W20260928 06:34:07.089023 26236 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:34:07.083707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:47295 (local address 127.16.157.126:40521)
0928 06:34:07.083857 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:07.083860 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:07.083870 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:34:07.083919 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:07.083920 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:07.083943 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:07.084037 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:07.084363 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:07.084888 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:07.085702 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:07.085977 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:07.086066 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:07.088951 (+ 2885us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.157.65:47295: BlockingRecv error: recv got EOF from 127.16.157.65:47295 (error 108)
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":67,"threads_started":1}
Sep 28 06:34:07 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.66@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:34:07 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.66@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 06:34:07.242446 26236 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:34:07.236721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.66:46419 (local address 127.16.157.126:40521)
0928 06:34:07.236783 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:07.236786 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:07.236857 (+ 71us) server_negotiation.cc:407] Connection header received
0928 06:34:07.236921 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:07.236923 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:07.236939 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:07.236996 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:07.237298 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:07.238368 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:07.239214 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:07.239598 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:07.239691 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:07.242400 (+ 2709us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.157.66:46419: BlockingRecv error: recv got EOF from 127.16.157.66:46419 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 06:34:07.242446 26237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:07.236644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40521 (local address 127.16.157.66:46419)
0928 06:34:07.236780 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:07.236795 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:07.236852 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:07.237018 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:07.237019 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:07.237180 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:34:07.237232 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:07.237233 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:07.238455 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:07.238456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:07.239130 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:07.239133 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:07.239686 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:07.239687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:07.239728 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:07.240240 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:34:07.240245 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:34:07.242313 (+ 2068us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.157.126:40521: Server oryx/127.16.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":85,"threads_started":1}
Sep 28 06:34:07 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.67@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:34:07 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.67@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:34:07.403080 26238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:07.396981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40521 (local address 127.16.157.67:38685)
0928 06:34:07.397119 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:07.397131 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:07.397237 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:07.397398 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:07.397399 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:07.397656 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:34:07.397704 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:07.397706 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:07.398697 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:07.398698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:07.399566 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:07.399568 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:07.400087 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:07.400087 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:07.400159 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:07.400711 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:34:07.400717 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:34:07.402971 (+ 2254us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.157.126:40521: Server oryx/127.16.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":69,"threads_started":1}
W20260928 06:34:07.403149 26236 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:34:07.397046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:38685 (local address 127.16.157.126:40521)
0928 06:34:07.397128 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:07.397131 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:07.397241 (+ 110us) server_negotiation.cc:407] Connection header received
0928 06:34:07.397308 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:07.397309 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:07.397324 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:07.397380 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:07.397784 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:07.398612 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:07.399654 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:07.399987 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:07.400059 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:07.403076 (+ 3017us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.157.67:38685: BlockingRecv error: recv got EOF from 127.16.157.67:38685 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 28 06:34:08 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.65@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:34:08 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.65@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:34:08.094941 26239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:08.089693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40521 (local address 127.16.157.65:36833)
0928 06:34:08.089828 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:08.089840 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:08.089963 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:08.090239 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:08.090240 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:08.090416 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:34:08.090462 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:08.090464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:08.091238 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:08.091239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:08.091833 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:08.091835 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:08.092219 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:08.092220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:08.092285 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:08.092815 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:34:08.092819 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:34:08.094849 (+ 2030us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.157.126:40521: Server oryx/127.16.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":62,"threads_started":1}
W20260928 06:34:08.094990 26240 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:34:08.089761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:36833 (local address 127.16.157.126:40521)
0928 06:34:08.089870 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:08.089873 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:08.089881 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:34:08.090083 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:08.090084 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:08.090121 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:08.090200 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:08.090546 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:08.091158 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:08.091936 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:08.092144 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:08.092221 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:08.094923 (+ 2702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.157.65:36833: BlockingRecv error: recv got EOF from 127.16.157.65:36833 (error 108)
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":48,"threads_started":1}
Sep 28 06:34:08 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.66@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:34:08 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.66@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:34:08.248343 26241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:08.243214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40521 (local address 127.16.157.66:51847)
0928 06:34:08.243360 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:08.243370 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:08.243425 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:08.243565 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:08.243566 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:08.243716 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:34:08.243765 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:08.243767 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:08.244695 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:08.244695 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:08.245201 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:08.245202 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:08.245717 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:08.245718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:08.245760 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:08.246267 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:34:08.246271 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:34:08.248284 (+ 2013us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.157.126:40521: Server oryx/127.16.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":62,"threads_started":1}
W20260928 06:34:08.248549 26240 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:34:08.243310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.66:51847 (local address 127.16.157.126:40521)
0928 06:34:08.243369 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:08.243371 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:08.243415 (+ 44us) server_negotiation.cc:407] Connection header received
0928 06:34:08.243475 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:08.243476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:08.243489 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:08.243553 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:08.243841 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:08.244616 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:08.245285 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:08.245623 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:08.245692 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:08.248412 (+ 2720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.157.66:51847: BlockingRecv error: recv got EOF from 127.16.157.66:51847 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 28 06:34:08 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.67@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:34:08 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.67@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:34:08.408774 26242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:08.403721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40521 (local address 127.16.157.67:49193)
0928 06:34:08.403891 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:08.403904 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:08.403975 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:08.404104 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:08.404105 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:08.404312 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:34:08.404373 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:08.404375 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:08.405017 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:08.405019 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:08.405954 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:08.405957 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:08.406293 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:08.406294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:08.406356 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:08.406792 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:34:08.406797 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:34:08.408718 (+ 1921us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.157.126:40521: Server oryx/127.16.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":88,"threads_started":1}
W20260928 06:34:08.408890 26240 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:34:08.403787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:49193 (local address 127.16.157.126:40521)
0928 06:34:08.403861 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:08.403863 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:08.403956 (+ 93us) server_negotiation.cc:407] Connection header received
0928 06:34:08.404025 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:08.404026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:08.404037 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:08.404079 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:08.404445 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:08.404927 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:08.406043 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:08.406222 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:08.406269 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:08.408828 (+ 2559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.157.67:49193: BlockingRecv error: recv got EOF from 127.16.157.67:49193 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 28 06:34:09 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.65@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:34:09 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.65@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:34:09.100705 26243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:09.095497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40521 (local address 127.16.157.65:60893)
0928 06:34:09.095641 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:09.095650 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:09.095698 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:09.095854 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:09.095854 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:09.096012 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:34:09.096060 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:09.096061 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:09.097023 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:09.097024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:09.097555 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:09.097556 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:09.098092 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:09.098092 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:09.098134 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:09.098635 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:34:09.098639 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:34:09.100646 (+ 2007us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.157.126:40521: Server oryx/127.16.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":67,"threads_started":1}
W20260928 06:34:09.100834 26244 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:34:09.095569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:60893 (local address 127.16.157.126:40521)
0928 06:34:09.095694 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:09.095696 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:09.095708 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:34:09.095740 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:09.095741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:09.095756 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:09.095851 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:09.096133 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:09.096933 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:09.097686 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:09.098011 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:09.098087 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:09.100762 (+ 2675us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.157.65:60893: BlockingRecv error: recv got EOF from 127.16.157.65:60893 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":61,"threads_started":1}
Sep 28 06:34:09 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.66@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:34:09 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.66@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:34:09.254067 26245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:09.248810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40521 (local address 127.16.157.66:55623)
0928 06:34:09.248974 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:09.248991 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:09.249043 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:09.249227 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:09.249228 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:09.249383 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:34:09.249435 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:09.249437 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:09.250576 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:09.250577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:09.251081 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:09.251082 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:09.251611 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:09.251612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:09.251649 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:09.252194 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:34:09.252199 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:34:09.254010 (+ 1811us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.157.126:40521: Server oryx/127.16.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":77,"threads_started":1}
W20260928 06:34:09.254279 26244 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:34:09.248922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.66:55623 (local address 127.16.157.126:40521)
0928 06:34:09.249001 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:09.249004 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:09.249037 (+ 33us) server_negotiation.cc:407] Connection header received
0928 06:34:09.249123 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:09.249124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:09.249139 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:09.249221 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:09.249521 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:09.250474 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:09.251167 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:09.251529 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:09.251612 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:09.254168 (+ 2556us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.157.66:55623: BlockingRecv error: recv got EOF from 127.16.157.66:55623 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 28 06:34:09 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.67@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:34:09 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.67@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:34:09.414506 26246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:09.409496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40521 (local address 127.16.157.67:33191)
0928 06:34:09.409643 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:09.409654 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:09.409724 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:09.409892 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:09.409893 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:09.410077 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:34:09.410123 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:09.410125 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:09.410810 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:09.410811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:09.411351 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:09.411353 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:09.411722 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:09.411723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:09.411785 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:09.412284 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:34:09.412289 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:34:09.414412 (+ 2123us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.157.126:40521: Server oryx/127.16.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":142,"threads_started":1}
W20260928 06:34:09.414547 26244 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:34:09.409543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:33191 (local address 127.16.157.126:40521)
0928 06:34:09.409619 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:09.409621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:09.409714 (+ 93us) server_negotiation.cc:407] Connection header received
0928 06:34:09.409788 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:09.409789 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:09.409801 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:09.409887 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:09.410216 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:09.410726 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:09.411439 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:09.411630 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:09.411721 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:09.414486 (+ 2765us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.157.67:33191: BlockingRecv error: recv got EOF from 127.16.157.67:33191 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 28 06:34:10 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.65@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:34:10 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.65@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:34:10.111091 26247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:10.101542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40521 (local address 127.16.157.65:33333)
0928 06:34:10.101724 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:10.101735 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:10.101788 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:10.101965 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:10.101966 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:10.102124 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:34:10.102175 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:10.102176 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:10.103294 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:10.103295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:10.103816 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:10.103817 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:10.104270 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:10.104271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:10.104310 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:10.108810 (+ 4500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:34:10.108815 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:34:10.111013 (+ 2198us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.157.126:40521: Server oryx/127.16.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":106,"threads_started":1}
W20260928 06:34:10.111246 26248 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:34:10.101617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:33333 (local address 127.16.157.126:40521)
0928 06:34:10.101755 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:10.101757 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:10.101774 (+ 17us) server_negotiation.cc:407] Connection header received
0928 06:34:10.101853 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:10.101854 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:10.101870 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:10.101952 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:10.102251 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:10.103208 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:10.103904 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:10.104195 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:10.104264 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:10.111184 (+ 6920us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.157.65:33333: BlockingRecv error: recv got EOF from 127.16.157.65:33333 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":65,"threads_started":1}
Sep 28 06:34:10 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.66@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:34:10 dist-test-slave-rrj5 krb5kdc[25181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.157.66@KRBTEST.COM for oryx/127.16.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:34:10.260282 26250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:10.254645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40521 (local address 127.16.157.66:58695)
0928 06:34:10.254768 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:10.254779 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:10.254843 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:10.254988 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:10.254989 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:10.255193 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:34:10.255243 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:10.255246 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:10.256289 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:10.256290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:10.256917 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:10.256920 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:10.257452 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:10.257453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:10.257493 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:10.258109 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:34:10.258114 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:34:10.260158 (+ 2044us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.157.126:40521: Server oryx/127.16.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":73,"threads_started":1}
W20260928 06:34:10.260365 26248 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:34:10.254714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.66:58695 (local address 127.16.157.126:40521)
0928 06:34:10.254762 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:10.254764 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:10.254829 (+ 65us) server_negotiation.cc:407] Connection header received
0928 06:34:10.254888 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:10.254889 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:10.254900 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:10.254963 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:10.255329 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:10.256198 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:10.257005 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:10.257365 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:10.257438 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:10.260286 (+ 2848us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.157.66:58695: BlockingRecv error: recv got EOF from 127.16.157.66:58695 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:34:10.317658 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 25767
I20260928 06:34:10.323189 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 25920
I20260928 06:34:10.327554 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 26055
I20260928 06:34:10.332710 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 25697
2026-09-28T06:34:10Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17688 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T06:34:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:34:10Z Disabled control of system clock
I20260928 06:34:10.362504 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577196946045-17013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577196946045-17013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577196946045-17013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:42705
--webserver_interface=127.16.157.126
--webserver_port=0
--builtin_ntp_servers=127.16.157.84:41881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.157.126:42705
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 06:34:10.452244 26254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:34:10.452546 26254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:34:10.452584 26254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:34:10.454648 26254 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:34:10.454713 26254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:34:10.454736 26254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:34:10.454761 26254 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:34:10.454780 26254 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:34:10.457042 26254 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:41881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577196946045-17013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577196946045-17013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.157.126:42705
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:42705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26254
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577196946045-17013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:34:10.457331 26254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:34:10.457733 26254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:34:10.462970 26262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:34:10.462980 26260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:34:10.462980 26259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:34:10.463551 26254 server_base.cc:1034] running on GCE node
I20260928 06:34:10.463806 26254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:34:10.464089 26254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:34:10.465284 26254 hybrid_clock.cc:648] HybridClock initialized: now 1790577250465274 us; error 44 us; skew 500 ppm
I20260928 06:34:10.466614 26254 webserver.cc:460] Webserver started at http://127.16.157.126:36059/ using document root <none> and password file <none>
I20260928 06:34:10.466874 26254 fs_manager.cc:359] Metadata directory not provided
I20260928 06:34:10.466938 26254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:34:10.467075 26254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:34:10.468106 26254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577196946045-17013-0/minicluster-data/master-0/data/instance:
uuid: "f89766d8a31143eb9abce879bb0f792a"
format_stamp: "Formatted at 2026-09-28 06:34:10 on dist-test-slave-rrj5"
server_key: "51674171c2eb241fa4e4523f28257e99"
server_key_iv: "63ad6ed1f115256f000e24299d54c224"
server_key_version: "encryptionkey@0"
I20260928 06:34:10.468467 26254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577196946045-17013-0/minicluster-data/master-0/wal/instance:
uuid: "f89766d8a31143eb9abce879bb0f792a"
format_stamp: "Formatted at 2026-09-28 06:34:10 on dist-test-slave-rrj5"
server_key: "51674171c2eb241fa4e4523f28257e99"
server_key_iv: "63ad6ed1f115256f000e24299d54c224"
server_key_version: "encryptionkey@0"
I20260928 06:34:10.469973 26254 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:34:10.470769 26268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:10.470937 26254 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:34:10.471035 26254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577196946045-17013-0/minicluster-data/master-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577196946045-17013-0/minicluster-data/master-0/wal
uuid: "f89766d8a31143eb9abce879bb0f792a"
format_stamp: "Formatted at 2026-09-28 06:34:10 on dist-test-slave-rrj5"
server_key: "51674171c2eb241fa4e4523f28257e99"
server_key_iv: "63ad6ed1f115256f000e24299d54c224"
server_key_version: "encryptionkey@0"
I20260928 06:34:10.471119 26254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577196946045-17013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577196946045-17013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577196946045-17013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:34:10.494863 26254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:34:10.495172 26254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:34:10.495314 26254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:34:10.499466 26254 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.126:42705
I20260928 06:34:10.499513 26320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.126:42705 every 8 connection(s)
I20260928 06:34:10.499831 26254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
I20260928 06:34:10.500516 26321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:34:10.503093 26321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f89766d8a31143eb9abce879bb0f792a: Bootstrap starting.
I20260928 06:34:10.503700 26321 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f89766d8a31143eb9abce879bb0f792a: Neither blocks nor log segments found. Creating new log.
I20260928 06:34:10.503991 26321 log.cc:824] T 00000000000000000000000000000000 P f89766d8a31143eb9abce879bb0f792a: Log is configured to *not* fsync() on all Append() calls
I20260928 06:34:10.504674 26321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f89766d8a31143eb9abce879bb0f792a: No bootstrap required, opened a new log
I20260928 06:34:10.506341 26321 raft_consensus.cc:348] T 00000000000000000000000000000000 P f89766d8a31143eb9abce879bb0f792a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f89766d8a31143eb9abce879bb0f792a" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 42705 } }
I20260928 06:34:10.506500 26321 raft_consensus.cc:374] T 00000000000000000000000000000000 P f89766d8a31143eb9abce879bb0f792a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:34:10.506546 26321 raft_consensus.cc:729] T 00000000000000000000000000000000 P f89766d8a31143eb9abce879bb0f792a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f89766d8a31143eb9abce879bb0f792a, State: Initialized, Role: FOLLOWER
I20260928 06:34:10.506731 26321 consensus_queue.cc:260] T 00000000000000000000000000000000 P f89766d8a31143eb9abce879bb0f792a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f89766d8a31143eb9abce879bb0f792a" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 42705 } }
I20260928 06:34:10.506832 26321 raft_consensus.cc:388] T 00000000000000000000000000000000 P f89766d8a31143eb9abce879bb0f792a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:34:10.506876 26321 raft_consensus.cc:482] T 00000000000000000000000000000000 P f89766d8a31143eb9abce879bb0f792a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:34:10.507059 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 26254
I20260928 06:34:10.507153 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577196946045-17013-0/minicluster-data/master-0/wal/instance
I20260928 06:34:10.507333 17013 external_mini_cluster.cc:1428] Setting key 7b4d6b5be8c10e358ece7815020f54b3
I20260928 06:34:10.507529 26321 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f89766d8a31143eb9abce879bb0f792a [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:34:10.508351 26321 raft_consensus.cc:504] T 00000000000000000000000000000000 P f89766d8a31143eb9abce879bb0f792a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f89766d8a31143eb9abce879bb0f792a" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 42705 } }
I20260928 06:34:10.508466 26321 leader_election.cc:302] T 00000000000000000000000000000000 P f89766d8a31143eb9abce879bb0f792a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f89766d8a31143eb9abce879bb0f792a; no voters:
I20260928 06:34:10.508622 26321 leader_election.cc:288] T 00000000000000000000000000000000 P f89766d8a31143eb9abce879bb0f792a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:34:10.508647 26326 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f89766d8a31143eb9abce879bb0f792a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:34:10.509115 26326 raft_consensus.cc:686] T 00000000000000000000000000000000 P f89766d8a31143eb9abce879bb0f792a [term 1 LEADER]: Becoming Leader. State: Replica: f89766d8a31143eb9abce879bb0f792a, State: Running, Role: LEADER
I20260928 06:34:10.509285 26326 consensus_queue.cc:237] T 00000000000000000000000000000000 P f89766d8a31143eb9abce879bb0f792a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f89766d8a31143eb9abce879bb0f792a" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 42705 } }
I20260928 06:34:10.508971 26321 sys_catalog.cc:564] T 00000000000000000000000000000000 P f89766d8a31143eb9abce879bb0f792a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:34:10.509625 26327 sys_catalog.cc:455] T 00000000000000000000000000000000 P f89766d8a31143eb9abce879bb0f792a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f89766d8a31143eb9abce879bb0f792a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f89766d8a31143eb9abce879bb0f792a" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 42705 } } }
I20260928 06:34:10.509707 26327 sys_catalog.cc:458] T 00000000000000000000000000000000 P f89766d8a31143eb9abce879bb0f792a [sys.catalog]: This master's current role is: LEADER
I20260928 06:34:10.509924 26330 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:34:10.509946 26328 sys_catalog.cc:455] T 00000000000000000000000000000000 P f89766d8a31143eb9abce879bb0f792a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f89766d8a31143eb9abce879bb0f792a. Latest consensus state: current_term: 1 leader_uuid: "f89766d8a31143eb9abce879bb0f792a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f89766d8a31143eb9abce879bb0f792a" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 42705 } } }
I20260928 06:34:10.510121 26328 sys_catalog.cc:458] T 00000000000000000000000000000000 P f89766d8a31143eb9abce879bb0f792a [sys.catalog]: This master's current role is: LEADER
I20260928 06:34:10.510401 26330 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:34:10.514115 26330 catalog_manager.cc:1347] Generated new cluster ID: e9e461f610bc4373af94a44a48ec4a58
I20260928 06:34:10.514173 26330 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:34:10.514380 26325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:10.507961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50396 (local address 127.16.157.126:42705)
0928 06:34:10.508167 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:10.508176 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:10.508334 (+ 158us) server_negotiation.cc:407] Connection header received
0928 06:34:10.508486 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:10.508494 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:10.508598 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:10.508750 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:10.509008 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:10.510802 (+ 1794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:10.511968 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:10.512661 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:10.512788 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:10.513522 (+ 734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:34:10.513526 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:34:10.513527 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:34:10.513760 (+ 233us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:34:10.513801 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:34:10.513815 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:34:10.513879 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:10.514078 (+ 199us) server_negotiation.cc:299] Negotiation successful
0928 06:34:10.514169 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":75,"threads_started":1}
I20260928 06:34:10.535876 26330 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:34:10.536350 26330 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:34:10.540134 26330 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f89766d8a31143eb9abce879bb0f792a: Generated new TSK 0
I20260928 06:34:10.540313 26330 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:34:10.542883 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.65:0
--local_ip_for_outbound_sockets=127.16.157.65
--webserver_interface=127.16.157.65
--webserver_port=0
--tserver_master_addrs=127.16.157.126:42705
--builtin_ntp_servers=127.16.157.84:41881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 06:34:10.654631 26345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:34:10.654960 26345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:34:10.655006 26345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:34:10.656508 26345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:34:10.656584 26345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:34:10.656635 26345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.65
I20260928 06:34:10.658316 26345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:41881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.157.65
--webserver_port=0
--tserver_master_addrs=127.16.157.126:42705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26345
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.65
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:34:10.658594 26345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:34:10.658921 26345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:34:10.665248 26350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:34:10.665262 26351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:34:10.665359 26353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:34:10.665809 26345 server_base.cc:1034] running on GCE node
I20260928 06:34:10.666054 26345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:34:10.666436 26345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:34:10.667613 26345 hybrid_clock.cc:648] HybridClock initialized: now 1790577250667588 us; error 39 us; skew 500 ppm
I20260928 06:34:10.669130 26345 webserver.cc:460] Webserver started at http://127.16.157.65:38727/ using document root <none> and password file <none>
I20260928 06:34:10.669416 26345 fs_manager.cc:359] Metadata directory not provided
I20260928 06:34:10.669461 26345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:34:10.669579 26345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:34:10.671025 26345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-0/data/instance:
uuid: "a8bea77e3555436f9a404058d23b06ee"
format_stamp: "Formatted at 2026-09-28 06:34:10 on dist-test-slave-rrj5"
server_key: "7feb31fb7fd8492c0c342e7a37249bed"
server_key_iv: "bbe379fb156b6c6de7ca197532c2f08d"
server_key_version: "encryptionkey@0"
I20260928 06:34:10.671460 26345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-0/wal/instance:
uuid: "a8bea77e3555436f9a404058d23b06ee"
format_stamp: "Formatted at 2026-09-28 06:34:10 on dist-test-slave-rrj5"
server_key: "7feb31fb7fd8492c0c342e7a37249bed"
server_key_iv: "bbe379fb156b6c6de7ca197532c2f08d"
server_key_version: "encryptionkey@0"
I20260928 06:34:10.673121 26345 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:34:10.674103 26359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:10.674296 26345 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:34:10.674394 26345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-0/wal
uuid: "a8bea77e3555436f9a404058d23b06ee"
format_stamp: "Formatted at 2026-09-28 06:34:10 on dist-test-slave-rrj5"
server_key: "7feb31fb7fd8492c0c342e7a37249bed"
server_key_iv: "bbe379fb156b6c6de7ca197532c2f08d"
server_key_version: "encryptionkey@0"
I20260928 06:34:10.674490 26345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:34:10.705041 26345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:34:10.705389 26345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:34:10.705559 26345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:34:10.705874 26345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:34:10.706348 26345 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:34:10.706390 26345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:10.706418 26345 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:34:10.706463 26345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:10.714355 26345 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.65:34343
I20260928 06:34:10.714390 26473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.65:34343 every 8 connection(s)
I20260928 06:34:10.714813 26345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
I20260928 06:34:10.719224 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 26345
I20260928 06:34:10.719331 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-0/wal/instance
I20260928 06:34:10.719551 17013 external_mini_cluster.cc:1428] Setting key 55c11bd155f26306261e04501d0eb1c7
I20260928 06:34:10.720644 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.66:0
--local_ip_for_outbound_sockets=127.16.157.66
--webserver_interface=127.16.157.66
--webserver_port=0
--tserver_master_addrs=127.16.157.126:42705
--builtin_ntp_servers=127.16.157.84:41881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 06:34:10.720997 26325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:10.715612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:46747 (local address 127.16.157.126:42705)
0928 06:34:10.715664 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:10.715667 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:10.716022 (+ 355us) server_negotiation.cc:407] Connection header received
0928 06:34:10.716902 (+ 880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:10.716903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:10.716918 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:10.716971 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:10.717708 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:10.718250 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:10.719314 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:10.719533 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:10.719613 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:10.720612 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:34:10.720614 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:34:10.720614 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:34:10.720622 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:34:10.720640 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:34:10.720646 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:34:10.720727 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:10.720843 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 06:34:10.720883 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:34:10.721231 26476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:10.715632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:42705 (local address 127.16.157.65:46747)
0928 06:34:10.715860 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:10.715916 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:10.716777 (+ 861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:10.717059 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:10.717062 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:10.717070 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:34:10.717556 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:10.717560 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:10.718355 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:10.718356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:10.719216 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:10.719218 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:10.719619 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:10.719621 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:10.719802 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:10.720434 (+ 632us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:34:10.720435 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:34:10.720474 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:34:10.720504 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:34:10.720741 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:34:10.720748 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:34:10.720752 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:34:10.720836 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 06:34:10.720964 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":92,"threads_started":1}
I20260928 06:34:10.721788 26474 heartbeater.cc:347] Connected to a master server at 127.16.157.126:42705
I20260928 06:34:10.721877 26474 heartbeater.cc:464] Registering TS with master...
I20260928 06:34:10.722139 26474 heartbeater.cc:511] Master 127.16.157.126:42705 requested a full tablet report, sending...
I20260928 06:34:10.722692 26285 ts_manager.cc:194] Registered new tserver with Master: a8bea77e3555436f9a404058d23b06ee (127.16.157.65:34343)
I20260928 06:34:10.723623 26285 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.157.65:46747
W20260928 06:34:10.813947 26477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:34:10.814147 26477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:34:10.814170 26477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:34:10.816237 26477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:34:10.816303 26477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:34:10.816354 26477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.66
I20260928 06:34:10.817785 26477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:41881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.157.66
--webserver_port=0
--tserver_master_addrs=127.16.157.126:42705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.66
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:34:10.818054 26477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:34:10.818353 26477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:34:10.823621 26485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:34:10.823627 26483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:34:10.823994 26482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:34:10.824275 26477 server_base.cc:1034] running on GCE node
I20260928 06:34:10.824501 26477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:34:10.824771 26477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:34:10.825945 26477 hybrid_clock.cc:648] HybridClock initialized: now 1790577250825935 us; error 30 us; skew 500 ppm
I20260928 06:34:10.827179 26477 webserver.cc:460] Webserver started at http://127.16.157.66:33487/ using document root <none> and password file <none>
I20260928 06:34:10.827433 26477 fs_manager.cc:359] Metadata directory not provided
I20260928 06:34:10.827502 26477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:34:10.827675 26477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:34:10.828846 26477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-1/data/instance:
uuid: "651223c80cbc48f7b179315a08e2b539"
format_stamp: "Formatted at 2026-09-28 06:34:10 on dist-test-slave-rrj5"
server_key: "de761e274863e3264641dd3138ec755c"
server_key_iv: "a496067544e5c506b695245bead7a9c4"
server_key_version: "encryptionkey@0"
I20260928 06:34:10.829221 26477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-1/wal/instance:
uuid: "651223c80cbc48f7b179315a08e2b539"
format_stamp: "Formatted at 2026-09-28 06:34:10 on dist-test-slave-rrj5"
server_key: "de761e274863e3264641dd3138ec755c"
server_key_iv: "a496067544e5c506b695245bead7a9c4"
server_key_version: "encryptionkey@0"
I20260928 06:34:10.831086 26477 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260928 06:34:10.831904 26491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:10.832105 26477 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:34:10.832216 26477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-1/wal
uuid: "651223c80cbc48f7b179315a08e2b539"
format_stamp: "Formatted at 2026-09-28 06:34:10 on dist-test-slave-rrj5"
server_key: "de761e274863e3264641dd3138ec755c"
server_key_iv: "a496067544e5c506b695245bead7a9c4"
server_key_version: "encryptionkey@0"
I20260928 06:34:10.832296 26477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:34:10.846542 26477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:34:10.846846 26477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:34:10.847015 26477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:34:10.847285 26477 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:34:10.847730 26477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:34:10.847788 26477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:10.847826 26477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:34:10.847857 26477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:10.855000 26477 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.66:36027
I20260928 06:34:10.855046 26605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.66:36027 every 8 connection(s)
I20260928 06:34:10.855489 26477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
I20260928 06:34:10.856261 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 26477
I20260928 06:34:10.856353 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-1/wal/instance
I20260928 06:34:10.856596 17013 external_mini_cluster.cc:1428] Setting key f45c340d6249c90c6c6bf71b12c65f76
I20260928 06:34:10.857661 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.67:0
--local_ip_for_outbound_sockets=127.16.157.67
--webserver_interface=127.16.157.67
--webserver_port=0
--tserver_master_addrs=127.16.157.126:42705
--builtin_ntp_servers=127.16.157.84:41881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 06:34:10.861809 26325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:10.856695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.66:36931 (local address 127.16.157.126:42705)
0928 06:34:10.856811 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:10.856813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:10.856828 (+ 15us) server_negotiation.cc:407] Connection header received
0928 06:34:10.857335 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:10.857336 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:10.857347 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:10.857405 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:10.858281 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:10.858783 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:10.860288 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:10.860497 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:10.860574 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:10.861455 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:34:10.861456 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:34:10.861457 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:34:10.861461 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:34:10.861475 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:34:10.861479 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:34:10.861533 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:10.861654 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 06:34:10.861687 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260928 06:34:10.862360 26608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:10.856356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:42705 (local address 127.16.157.66:36931)
0928 06:34:10.856574 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:10.856610 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:10.857241 (+ 631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:10.857544 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:10.857548 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:10.857559 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:34:10.858183 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:10.858189 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:10.858880 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:10.858881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:10.860216 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:10.860219 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:10.860579 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:10.860580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:10.860719 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:10.861274 (+ 555us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:34:10.861278 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:34:10.861307 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:34:10.861346 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:34:10.861536 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:34:10.861541 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:34:10.861545 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:34:10.861965 (+ 420us) client_negotiation.cc:240] Negotiation successful
0928 06:34:10.862073 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":118,"threads_started":1}
I20260928 06:34:10.862998 26606 heartbeater.cc:347] Connected to a master server at 127.16.157.126:42705
I20260928 06:34:10.863104 26606 heartbeater.cc:464] Registering TS with master...
I20260928 06:34:10.863307 26606 heartbeater.cc:511] Master 127.16.157.126:42705 requested a full tablet report, sending...
I20260928 06:34:10.863790 26285 ts_manager.cc:194] Registered new tserver with Master: 651223c80cbc48f7b179315a08e2b539 (127.16.157.66:36027)
I20260928 06:34:10.864321 26285 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.157.66:36931
W20260928 06:34:10.957696 26609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:34:10.957892 26609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:34:10.957918 26609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:34:10.959595 26609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:34:10.959659 26609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:34:10.959709 26609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.67
I20260928 06:34:10.961270 26609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:41881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.157.67
--webserver_port=0
--tserver_master_addrs=127.16.157.126:42705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.67
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:34:10.961555 26609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:34:10.961896 26609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:34:10.967370 26615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:34:10.967398 26614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:34:10.967427 26617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:34:10.967583 26609 server_base.cc:1034] running on GCE node
I20260928 06:34:10.967962 26609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:34:10.968261 26609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:34:10.969439 26609 hybrid_clock.cc:648] HybridClock initialized: now 1790577250969413 us; error 37 us; skew 500 ppm
I20260928 06:34:10.970834 26609 webserver.cc:460] Webserver started at http://127.16.157.67:38921/ using document root <none> and password file <none>
I20260928 06:34:10.971155 26609 fs_manager.cc:359] Metadata directory not provided
I20260928 06:34:10.971236 26609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:34:10.971367 26609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:34:10.972882 26609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-2/data/instance:
uuid: "c18d48bfb3a64b1cb584806bdfa6516b"
format_stamp: "Formatted at 2026-09-28 06:34:10 on dist-test-slave-rrj5"
server_key: "f26c30e930bb31d1a78ce31f107912e5"
server_key_iv: "8bfc2f7394f36e6f7f316e4ef243da85"
server_key_version: "encryptionkey@0"
I20260928 06:34:10.973253 26609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-2/wal/instance:
uuid: "c18d48bfb3a64b1cb584806bdfa6516b"
format_stamp: "Formatted at 2026-09-28 06:34:10 on dist-test-slave-rrj5"
server_key: "f26c30e930bb31d1a78ce31f107912e5"
server_key_iv: "8bfc2f7394f36e6f7f316e4ef243da85"
server_key_version: "encryptionkey@0"
I20260928 06:34:10.974941 26609 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260928 06:34:10.975893 26623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:10.976100 26609 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:34:10.976192 26609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-2/wal
uuid: "c18d48bfb3a64b1cb584806bdfa6516b"
format_stamp: "Formatted at 2026-09-28 06:34:10 on dist-test-slave-rrj5"
server_key: "f26c30e930bb31d1a78ce31f107912e5"
server_key_iv: "8bfc2f7394f36e6f7f316e4ef243da85"
server_key_version: "encryptionkey@0"
I20260928 06:34:10.976263 26609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:34:10.995738 26609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:34:10.996031 26609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:34:10.996176 26609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:34:10.996395 26609 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:34:10.996722 26609 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:34:10.996775 26609 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:10.996817 26609 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:34:10.996845 26609 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:11.003777 26609 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.67:43255
I20260928 06:34:11.003820 26736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.67:43255 every 8 connection(s)
I20260928 06:34:11.004142 26609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
I20260928 06:34:11.010562 26325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:11.004896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:39763 (local address 127.16.157.126:42705)
0928 06:34:11.004948 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:11.004950 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:11.005250 (+ 300us) server_negotiation.cc:407] Connection header received
0928 06:34:11.005881 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:11.005883 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:11.005898 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:11.005976 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:11.006624 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:11.007140 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:11.008788 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:11.009007 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:11.009097 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:11.010274 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:34:11.010275 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:34:11.010276 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:34:11.010280 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:34:11.010294 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:34:11.010299 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:34:11.010370 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:11.010451 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 06:34:11.010481 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 06:34:11.010830 26739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:11.004887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:42705 (local address 127.16.157.67:39763)
0928 06:34:11.005089 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:11.005136 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:11.005777 (+ 641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:11.006028 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:11.006031 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:11.006040 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:34:11.006533 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:11.006538 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:11.007237 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:11.007239 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:11.008682 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:11.008685 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:11.009107 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:11.009108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:11.009256 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:11.010104 (+ 848us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:34:11.010104 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:34:11.010136 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:34:11.010164 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:34:11.010370 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:34:11.010376 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:34:11.010380 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:34:11.010453 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 06:34:11.010554 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":71,"threads_started":1}
I20260928 06:34:11.011345 26737 heartbeater.cc:347] Connected to a master server at 127.16.157.126:42705
I20260928 06:34:11.011425 26737 heartbeater.cc:464] Registering TS with master...
I20260928 06:34:11.011637 26737 heartbeater.cc:511] Master 127.16.157.126:42705 requested a full tablet report, sending...
I20260928 06:34:11.012082 26285 ts_manager.cc:194] Registered new tserver with Master: c18d48bfb3a64b1cb584806bdfa6516b (127.16.157.67:43255)
I20260928 06:34:11.012607 26285 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.157.67:39763
I20260928 06:34:11.013957 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 26609
I20260928 06:34:11.014072 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-2/wal/instance
I20260928 06:34:11.014281 17013 external_mini_cluster.cc:1428] Setting key d8461ac31a911bfb8da6c9353a5338cf
I20260928 06:34:11.015201 17013 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 06:34:11.017417 26325 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:34:11.016854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50398 (local address 127.16.157.126:42705)
0928 06:34:11.016901 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:11.016903 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:11.017015 (+ 112us) server_negotiation.cc:407] Connection header received
0928 06:34:11.017099 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:11.017100 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:11.017110 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:11.017194 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:11.017340 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50398: BlockingRecv error: recv got EOF from 127.0.0.1:50398 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 06:34:11.017467 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 26345
I20260928 06:34:11.023340 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 26477
I20260928 06:34:11.028481 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 26609
I20260928 06:34:11.033367 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 26254
2026-09-28T06:34:11Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (698 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T06:34:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:34:11Z Disabled control of system clock
I20260928 06:34:11.061928 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577196946045-17013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577196946045-17013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577196946045-17013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:34089
--webserver_interface=127.16.157.126
--webserver_port=0
--builtin_ntp_servers=127.16.157.84:40331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.157.126:34089
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 06:34:11.150240 26748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:34:11.150553 26748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:34:11.150609 26748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:34:11.152601 26748 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:34:11.152660 26748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:34:11.152684 26748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:34:11.152709 26748 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:34:11.152738 26748 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:34:11.155053 26748 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:40331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577196946045-17013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577196946045-17013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.157.126:34089
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:34089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577196946045-17013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:34:11.155321 26748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:34:11.155699 26748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:34:11.161037 26756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:34:11.161005 26754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:34:11.161185 26753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:34:11.161365 26748 server_base.cc:1034] running on GCE node
I20260928 06:34:11.161681 26748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:34:11.162067 26748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:34:11.163283 26748 hybrid_clock.cc:648] HybridClock initialized: now 1790577251163262 us; error 40 us; skew 500 ppm
I20260928 06:34:11.164948 26748 webserver.cc:460] Webserver started at http://127.16.157.126:38229/ using document root <none> and password file <none>
I20260928 06:34:11.165215 26748 fs_manager.cc:359] Metadata directory not provided
I20260928 06:34:11.165266 26748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:34:11.165431 26748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:34:11.167028 26748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577196946045-17013-0/minicluster-data/master-0/data/instance:
uuid: "880d7f71b27d4953bcf9934fc50cd47f"
format_stamp: "Formatted at 2026-09-28 06:34:11 on dist-test-slave-rrj5"
server_key: "fceb04c1cca445a1c4878315386ab4c8"
server_key_iv: "b321ab184178b4d29703e5f7c3f03ef2"
server_key_version: "encryptionkey@0"
I20260928 06:34:11.167510 26748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577196946045-17013-0/minicluster-data/master-0/wal/instance:
uuid: "880d7f71b27d4953bcf9934fc50cd47f"
format_stamp: "Formatted at 2026-09-28 06:34:11 on dist-test-slave-rrj5"
server_key: "fceb04c1cca445a1c4878315386ab4c8"
server_key_iv: "b321ab184178b4d29703e5f7c3f03ef2"
server_key_version: "encryptionkey@0"
I20260928 06:34:11.169595 26748 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260928 06:34:11.170545 26762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:11.170733 26748 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:34:11.170843 26748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577196946045-17013-0/minicluster-data/master-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577196946045-17013-0/minicluster-data/master-0/wal
uuid: "880d7f71b27d4953bcf9934fc50cd47f"
format_stamp: "Formatted at 2026-09-28 06:34:11 on dist-test-slave-rrj5"
server_key: "fceb04c1cca445a1c4878315386ab4c8"
server_key_iv: "b321ab184178b4d29703e5f7c3f03ef2"
server_key_version: "encryptionkey@0"
I20260928 06:34:11.170930 26748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577196946045-17013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577196946045-17013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577196946045-17013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:34:11.176112 26748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:34:11.176460 26748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:34:11.176617 26748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:34:11.181676 26748 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.126:34089
I20260928 06:34:11.181749 26814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.126:34089 every 8 connection(s)
I20260928 06:34:11.182113 26748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
I20260928 06:34:11.182819 26815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:34:11.185527 26815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 880d7f71b27d4953bcf9934fc50cd47f: Bootstrap starting.
I20260928 06:34:11.186229 26815 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 880d7f71b27d4953bcf9934fc50cd47f: Neither blocks nor log segments found. Creating new log.
I20260928 06:34:11.186318 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 26748
I20260928 06:34:11.186421 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577196946045-17013-0/minicluster-data/master-0/wal/instance
I20260928 06:34:11.186616 17013 external_mini_cluster.cc:1428] Setting key d6c12eebe68e6f8beeada93f12409ee2
I20260928 06:34:11.186602 26815 log.cc:824] T 00000000000000000000000000000000 P 880d7f71b27d4953bcf9934fc50cd47f: Log is configured to *not* fsync() on all Append() calls
I20260928 06:34:11.187616 26815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 880d7f71b27d4953bcf9934fc50cd47f: No bootstrap required, opened a new log
I20260928 06:34:11.189854 26819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:11.187182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58300 (local address 127.16.157.126:34089)
0928 06:34:11.187358 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:11.187367 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:11.187559 (+ 192us) server_negotiation.cc:407] Connection header received
0928 06:34:11.188329 (+ 770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:11.188340 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:11.188395 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:11.188555 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:11.189303 (+ 748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:34:11.189306 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:34:11.189307 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:34:11.189312 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:34:11.189363 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:34:11.189374 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:34:11.189493 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:11.189550 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 06:34:11.189664 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":79,"threads_started":1}
I20260928 06:34:11.190407 26815 raft_consensus.cc:348] T 00000000000000000000000000000000 P 880d7f71b27d4953bcf9934fc50cd47f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "880d7f71b27d4953bcf9934fc50cd47f" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 34089 } }
I20260928 06:34:11.190632 26815 raft_consensus.cc:374] T 00000000000000000000000000000000 P 880d7f71b27d4953bcf9934fc50cd47f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:34:11.190706 26815 raft_consensus.cc:729] T 00000000000000000000000000000000 P 880d7f71b27d4953bcf9934fc50cd47f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 880d7f71b27d4953bcf9934fc50cd47f, State: Initialized, Role: FOLLOWER
I20260928 06:34:11.190837 26815 consensus_queue.cc:260] T 00000000000000000000000000000000 P 880d7f71b27d4953bcf9934fc50cd47f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "880d7f71b27d4953bcf9934fc50cd47f" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 34089 } }
I20260928 06:34:11.190927 26815 raft_consensus.cc:388] T 00000000000000000000000000000000 P 880d7f71b27d4953bcf9934fc50cd47f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:34:11.190975 26815 raft_consensus.cc:482] T 00000000000000000000000000000000 P 880d7f71b27d4953bcf9934fc50cd47f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:34:11.191035 26815 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 880d7f71b27d4953bcf9934fc50cd47f [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:34:11.191908 26815 raft_consensus.cc:504] T 00000000000000000000000000000000 P 880d7f71b27d4953bcf9934fc50cd47f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "880d7f71b27d4953bcf9934fc50cd47f" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 34089 } }
I20260928 06:34:11.192049 26815 leader_election.cc:302] T 00000000000000000000000000000000 P 880d7f71b27d4953bcf9934fc50cd47f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 880d7f71b27d4953bcf9934fc50cd47f; no voters:
I20260928 06:34:11.192220 26815 leader_election.cc:288] T 00000000000000000000000000000000 P 880d7f71b27d4953bcf9934fc50cd47f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:34:11.192277 26820 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 880d7f71b27d4953bcf9934fc50cd47f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:34:11.192477 26820 raft_consensus.cc:686] T 00000000000000000000000000000000 P 880d7f71b27d4953bcf9934fc50cd47f [term 1 LEADER]: Becoming Leader. State: Replica: 880d7f71b27d4953bcf9934fc50cd47f, State: Running, Role: LEADER
I20260928 06:34:11.192507 26815 sys_catalog.cc:564] T 00000000000000000000000000000000 P 880d7f71b27d4953bcf9934fc50cd47f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:34:11.192655 26820 consensus_queue.cc:237] T 00000000000000000000000000000000 P 880d7f71b27d4953bcf9934fc50cd47f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "880d7f71b27d4953bcf9934fc50cd47f" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 34089 } }
I20260928 06:34:11.193107 26821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 880d7f71b27d4953bcf9934fc50cd47f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "880d7f71b27d4953bcf9934fc50cd47f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "880d7f71b27d4953bcf9934fc50cd47f" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 34089 } } }
I20260928 06:34:11.193259 26821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 880d7f71b27d4953bcf9934fc50cd47f [sys.catalog]: This master's current role is: LEADER
I20260928 06:34:11.193117 26822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 880d7f71b27d4953bcf9934fc50cd47f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 880d7f71b27d4953bcf9934fc50cd47f. Latest consensus state: current_term: 1 leader_uuid: "880d7f71b27d4953bcf9934fc50cd47f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "880d7f71b27d4953bcf9934fc50cd47f" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 34089 } } }
I20260928 06:34:11.193328 26822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 880d7f71b27d4953bcf9934fc50cd47f [sys.catalog]: This master's current role is: LEADER
I20260928 06:34:11.193485 26829 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:34:11.193984 26829 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:34:11.195536 26829 catalog_manager.cc:1347] Generated new cluster ID: a52f2803b80e44c7b4c9e267d9c3b021
I20260928 06:34:11.195593 26829 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:34:11.205860 26829 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:34:11.206023 26829 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:34:11.214685 26829 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 880d7f71b27d4953bcf9934fc50cd47f: Generated new TSK 0
I20260928 06:34:11.214896 26829 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:34:11.222520 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.65:0
--local_ip_for_outbound_sockets=127.16.157.65
--webserver_interface=127.16.157.65
--webserver_port=0
--tserver_master_addrs=127.16.157.126:34089
--builtin_ntp_servers=127.16.157.84:40331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 06:34:11.341996 26839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:34:11.342262 26839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:34:11.342294 26839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:34:11.344094 26839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:34:11.344151 26839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:34:11.344190 26839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.65
I20260928 06:34:11.346269 26839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:40331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.157.65
--webserver_port=0
--tserver_master_addrs=127.16.157.126:34089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.65
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:34:11.346531 26839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:34:11.346899 26839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:34:11.353225 26845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:34:11.353240 26847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:34:11.353492 26844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:34:11.354426 26839 server_base.cc:1034] running on GCE node
I20260928 06:34:11.354681 26839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:34:11.354933 26839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:34:11.356099 26839 hybrid_clock.cc:648] HybridClock initialized: now 1790577251356068 us; error 44 us; skew 500 ppm
I20260928 06:34:11.357739 26839 webserver.cc:460] Webserver started at http://127.16.157.65:38579/ using document root <none> and password file <none>
I20260928 06:34:11.358065 26839 fs_manager.cc:359] Metadata directory not provided
I20260928 06:34:11.358135 26839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:34:11.358275 26839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:34:11.359443 26839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-0/data/instance:
uuid: "5925c7e903c94fb49751bb9d1d6053da"
format_stamp: "Formatted at 2026-09-28 06:34:11 on dist-test-slave-rrj5"
server_key: "52ab5b886df67b5655dc940ea4ca5c11"
server_key_iv: "8d3497363857c2e762644dac1e1c4dab"
server_key_version: "encryptionkey@0"
I20260928 06:34:11.359850 26839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-0/wal/instance:
uuid: "5925c7e903c94fb49751bb9d1d6053da"
format_stamp: "Formatted at 2026-09-28 06:34:11 on dist-test-slave-rrj5"
server_key: "52ab5b886df67b5655dc940ea4ca5c11"
server_key_iv: "8d3497363857c2e762644dac1e1c4dab"
server_key_version: "encryptionkey@0"
I20260928 06:34:11.361927 26839 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:34:11.363953 26853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:11.364192 26839 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 06:34:11.364287 26839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-0/wal
uuid: "5925c7e903c94fb49751bb9d1d6053da"
format_stamp: "Formatted at 2026-09-28 06:34:11 on dist-test-slave-rrj5"
server_key: "52ab5b886df67b5655dc940ea4ca5c11"
server_key_iv: "8d3497363857c2e762644dac1e1c4dab"
server_key_version: "encryptionkey@0"
I20260928 06:34:11.364363 26839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:34:11.368315 26839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:34:11.368597 26839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:34:11.368752 26839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:34:11.368973 26839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:34:11.369289 26839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:34:11.369345 26839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:11.369383 26839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:34:11.369406 26839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:11.376155 26839 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.65:34605
I20260928 06:34:11.376207 26967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.65:34605 every 8 connection(s)
I20260928 06:34:11.376801 26839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
I20260928 06:34:11.378373 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 26839
I20260928 06:34:11.378460 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-0/wal/instance
I20260928 06:34:11.378648 17013 external_mini_cluster.cc:1428] Setting key 788171a247dc517c7ff6be248ee0763b
I20260928 06:34:11.379688 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.66:0
--local_ip_for_outbound_sockets=127.16.157.66
--webserver_interface=127.16.157.66
--webserver_port=0
--tserver_master_addrs=127.16.157.126:34089
--builtin_ntp_servers=127.16.157.84:40331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 06:34:11.380810 26819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:11.377565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:53207 (local address 127.16.157.126:34089)
0928 06:34:11.377638 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:11.377640 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:11.377940 (+ 300us) server_negotiation.cc:407] Connection header received
0928 06:34:11.378735 (+ 795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:11.378737 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:11.378744 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:11.378778 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:11.380494 (+ 1716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:34:11.380496 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:34:11.380496 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:34:11.380501 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:34:11.380518 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:34:11.380522 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:34:11.380554 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:11.380698 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 06:34:11.380715 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:34:11.381286 26970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:11.377554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:34089 (local address 127.16.157.65:53207)
0928 06:34:11.377775 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:11.377827 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:11.378584 (+ 757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:11.379000 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:11.379004 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:11.379015 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:34:11.380289 (+ 1274us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:34:11.380290 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:34:11.380336 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:34:11.380380 (+ 44us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:34:11.380600 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:34:11.380607 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:34:11.380610 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:34:11.380940 (+ 330us) client_negotiation.cc:240] Negotiation successful
0928 06:34:11.381016 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":83,"threads_started":1}
I20260928 06:34:11.381937 26968 heartbeater.cc:347] Connected to a master server at 127.16.157.126:34089
I20260928 06:34:11.382074 26968 heartbeater.cc:464] Registering TS with master...
I20260928 06:34:11.382254 26968 heartbeater.cc:511] Master 127.16.157.126:34089 requested a full tablet report, sending...
I20260928 06:34:11.382841 26779 ts_manager.cc:194] Registered new tserver with Master: 5925c7e903c94fb49751bb9d1d6053da (127.16.157.65:34605)
W20260928 06:34:11.472640 26971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:34:11.472931 26971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:34:11.472975 26971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:34:11.474191 26971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:34:11.474231 26971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:34:11.474257 26971 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.66
I20260928 06:34:11.476374 26971 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:40331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.157.66
--webserver_port=0
--tserver_master_addrs=127.16.157.126:34089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.66
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:34:11.476655 26971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:34:11.476938 26971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:34:11.482251 26976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:34:11.482409 26979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:34:11.482512 26977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:34:11.482632 26971 server_base.cc:1034] running on GCE node
I20260928 06:34:11.482861 26971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:34:11.483160 26971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:34:11.484349 26971 hybrid_clock.cc:648] HybridClock initialized: now 1790577251484323 us; error 31 us; skew 500 ppm
I20260928 06:34:11.486084 26971 webserver.cc:460] Webserver started at http://127.16.157.66:33075/ using document root <none> and password file <none>
I20260928 06:34:11.486356 26971 fs_manager.cc:359] Metadata directory not provided
I20260928 06:34:11.486424 26971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:34:11.486589 26971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:34:11.487739 26971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-1/data/instance:
uuid: "ad1e5240ff424bc59049726da0f3705f"
format_stamp: "Formatted at 2026-09-28 06:34:11 on dist-test-slave-rrj5"
server_key: "cc6327e61375d18a6df1b680ed4b3116"
server_key_iv: "26cbd9eb81bdbce6e7a469d0fff2cc49"
server_key_version: "encryptionkey@0"
I20260928 06:34:11.488095 26971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-1/wal/instance:
uuid: "ad1e5240ff424bc59049726da0f3705f"
format_stamp: "Formatted at 2026-09-28 06:34:11 on dist-test-slave-rrj5"
server_key: "cc6327e61375d18a6df1b680ed4b3116"
server_key_iv: "26cbd9eb81bdbce6e7a469d0fff2cc49"
server_key_version: "encryptionkey@0"
I20260928 06:34:11.489636 26971 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:34:11.490564 26985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:11.490748 26971 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:34:11.490847 26971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-1/wal
uuid: "ad1e5240ff424bc59049726da0f3705f"
format_stamp: "Formatted at 2026-09-28 06:34:11 on dist-test-slave-rrj5"
server_key: "cc6327e61375d18a6df1b680ed4b3116"
server_key_iv: "26cbd9eb81bdbce6e7a469d0fff2cc49"
server_key_version: "encryptionkey@0"
I20260928 06:34:11.490935 26971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:34:11.494809 26971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:34:11.495164 26971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:34:11.495316 26971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:34:11.495559 26971 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:34:11.495864 26971 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:34:11.495898 26971 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:11.495919 26971 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:34:11.495931 26971 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:11.502223 26971 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.66:40151
I20260928 06:34:11.502357 27100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.66:40151 every 8 connection(s)
I20260928 06:34:11.502602 26971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
I20260928 06:34:11.505750 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 26971
I20260928 06:34:11.505879 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-1/wal/instance
I20260928 06:34:11.506078 17013 external_mini_cluster.cc:1428] Setting key e6490dcc395ffba047db9caac7611b3c
I20260928 06:34:11.506444 26819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:11.503338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.66:37001 (local address 127.16.157.126:34089)
0928 06:34:11.503420 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:11.503423 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:11.503698 (+ 275us) server_negotiation.cc:407] Connection header received
0928 06:34:11.504208 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:11.504210 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:11.504216 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:11.504267 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:11.506100 (+ 1833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:34:11.506102 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:34:11.506102 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:34:11.506107 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:34:11.506125 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:34:11.506130 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:34:11.506183 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:11.506322 (+ 139us) server_negotiation.cc:299] Negotiation successful
0928 06:34:11.506340 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 06:34:11.506990 27103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:11.503327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:34089 (local address 127.16.157.66:37001)
0928 06:34:11.503554 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:11.503596 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:11.504096 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:11.504334 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:11.504337 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:11.504346 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:34:11.505910 (+ 1564us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:34:11.505910 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:34:11.505951 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:34:11.505987 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:34:11.506215 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:34:11.506221 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:34:11.506223 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:34:11.506323 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 06:34:11.506388 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":102,"threads_started":1}
I20260928 06:34:11.507192 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.67:0
--local_ip_for_outbound_sockets=127.16.157.67
--webserver_interface=127.16.157.67
--webserver_port=0
--tserver_master_addrs=127.16.157.126:34089
--builtin_ntp_servers=127.16.157.84:40331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 06:34:11.509239 27101 heartbeater.cc:347] Connected to a master server at 127.16.157.126:34089
I20260928 06:34:11.509382 27101 heartbeater.cc:464] Registering TS with master...
I20260928 06:34:11.509611 27101 heartbeater.cc:511] Master 127.16.157.126:34089 requested a full tablet report, sending...
I20260928 06:34:11.510121 26779 ts_manager.cc:194] Registered new tserver with Master: ad1e5240ff424bc59049726da0f3705f (127.16.157.66:40151)
W20260928 06:34:11.600530 27104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:34:11.600795 27104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:34:11.600837 27104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:34:11.602919 27104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:34:11.602979 27104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:34:11.603040 27104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.67
I20260928 06:34:11.605230 27104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:40331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.157.67
--webserver_port=0
--tserver_master_addrs=127.16.157.126:34089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.67
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:34:11.605477 27104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:34:11.605842 27104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:34:11.611230 27110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:34:11.611234 27109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:34:11.611543 27112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:34:11.611671 27104 server_base.cc:1034] running on GCE node
I20260928 06:34:11.611893 27104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:34:11.612157 27104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:34:11.613315 27104 hybrid_clock.cc:648] HybridClock initialized: now 1790577251613298 us; error 35 us; skew 500 ppm
I20260928 06:34:11.614686 27104 webserver.cc:460] Webserver started at http://127.16.157.67:39455/ using document root <none> and password file <none>
I20260928 06:34:11.614939 27104 fs_manager.cc:359] Metadata directory not provided
I20260928 06:34:11.615005 27104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:34:11.615157 27104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:34:11.616410 27104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-2/data/instance:
uuid: "0f12df38861f44968cc346a7b1694a15"
format_stamp: "Formatted at 2026-09-28 06:34:11 on dist-test-slave-rrj5"
server_key: "c1f76749598c068d115eec343d96c7a7"
server_key_iv: "105cad1282ce36fd6085b111df083815"
server_key_version: "encryptionkey@0"
I20260928 06:34:11.616775 27104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-2/wal/instance:
uuid: "0f12df38861f44968cc346a7b1694a15"
format_stamp: "Formatted at 2026-09-28 06:34:11 on dist-test-slave-rrj5"
server_key: "c1f76749598c068d115eec343d96c7a7"
server_key_iv: "105cad1282ce36fd6085b111df083815"
server_key_version: "encryptionkey@0"
I20260928 06:34:11.618383 27104 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:34:11.619196 27118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:11.619498 27104 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:34:11.619592 27104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-2/wal
uuid: "0f12df38861f44968cc346a7b1694a15"
format_stamp: "Formatted at 2026-09-28 06:34:11 on dist-test-slave-rrj5"
server_key: "c1f76749598c068d115eec343d96c7a7"
server_key_iv: "105cad1282ce36fd6085b111df083815"
server_key_version: "encryptionkey@0"
I20260928 06:34:11.619664 27104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:34:11.622975 27104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:34:11.623215 27104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:34:11.623351 27104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:34:11.623550 27104 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:34:11.623874 27104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:34:11.623926 27104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:11.623960 27104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:34:11.623983 27104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:11.630378 27104 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.67:43423
I20260928 06:34:11.630420 27233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.67:43423 every 8 connection(s)
I20260928 06:34:11.630715 27104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
I20260928 06:34:11.632723 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 27104
I20260928 06:34:11.632817 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577196946045-17013-0/minicluster-data/ts-2/wal/instance
I20260928 06:34:11.633010 17013 external_mini_cluster.cc:1428] Setting key ebdd4d6373a62ca73b74c61e17bced8d
I20260928 06:34:11.634266 26819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:11.631500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:53827 (local address 127.16.157.126:34089)
0928 06:34:11.631546 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:11.631548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:11.631872 (+ 324us) server_negotiation.cc:407] Connection header received
0928 06:34:11.632337 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:11.632339 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:11.632347 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:11.632419 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:11.634055 (+ 1636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:34:11.634056 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:34:11.634056 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:34:11.634060 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:34:11.634075 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:34:11.634079 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:34:11.634122 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:11.634204 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 06:34:11.634215 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:34:11.634701 27236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:11.631497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:34089 (local address 127.16.157.67:53827)
0928 06:34:11.631713 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:11.631760 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:11.632244 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:11.632497 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:11.632500 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:11.632511 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:34:11.633883 (+ 1372us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:34:11.633884 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:34:11.633921 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:34:11.633956 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:34:11.634125 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:34:11.634130 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:34:11.634133 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:34:11.634217 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 06:34:11.634283 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":99,"threads_started":1}
I20260928 06:34:11.635149 27234 heartbeater.cc:347] Connected to a master server at 127.16.157.126:34089
I20260928 06:34:11.635232 27234 heartbeater.cc:464] Registering TS with master...
I20260928 06:34:11.635392 27234 heartbeater.cc:511] Master 127.16.157.126:34089 requested a full tablet report, sending...
I20260928 06:34:11.635792 26779 ts_manager.cc:194] Registered new tserver with Master: 0f12df38861f44968cc346a7b1694a15 (127.16.157.67:43423)
I20260928 06:34:11.644698 17013 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 06:34:11.646819 26819 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:34:11.646272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58306 (local address 127.16.157.126:34089)
0928 06:34:11.646344 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:11.646346 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:11.646476 (+ 130us) server_negotiation.cc:407] Connection header received
0928 06:34:11.646544 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:11.646545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:11.646549 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:11.646602 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:11.646746 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58306: BlockingRecv error: recv got EOF from 127.0.0.1:58306 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 06:34:11.646867 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 26839
I20260928 06:34:11.652490 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 26971
I20260928 06:34:11.657198 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 27104
I20260928 06:34:11.661511 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 26748
2026-09-28T06:34:11Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (627 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:34:11 dist-test-slave-rrj5 krb5kdc[27244](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:34:11 dist-test-slave-rrj5 krb5kdc[27244](info): set up 2 sockets
Sep 28 06:34:11 dist-test-slave-rrj5 krb5kdc[27244](info): commencing operation
krb5kdc: starting...
W20260928 06:34:12.771884 17013 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Sep 28 06:34:12 dist-test-slave-rrj5 krb5kdc[27244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:34:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:34:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.157.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab.
Entry for principal kudu/127.16.157.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab.
I20260928 06:34:12.909220 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:46459
--webserver_interface=127.16.157.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=kudu/127.16.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:41133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.157.126:46459
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:34:13.015123 27257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:34:13.015460 27257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:34:13.015529 27257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:34:13.017612 27257 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:34:13.017683 27257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:34:13.017724 27257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:34:13.017771 27257 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:34:13.017818 27257 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:34:13.020181 27257 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:41133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.157.126:46459
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=kudu/127.16.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:46459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:34:13.020480 27257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:34:13.020736 27257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:34:13.026201 27263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:34:13.026338 27262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:34:13.026206 27265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:34:13.026870 27257 server_base.cc:1034] running on GCE node
I20260928 06:34:13.027174 27257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:34:13.027470 27257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:34:13.028644 27257 hybrid_clock.cc:648] HybridClock initialized: now 1790577253028631 us; error 36 us; skew 500 ppm
Sep 28 06:34:13 dist-test-slave-rrj5 krb5kdc[27244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577253, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:34:13.031594 27257 init.cc:377] Logged in from keytab as kudu/127.16.157.126@KRBTEST.COM (short username kudu)
I20260928 06:34:13.032223 27257 webserver.cc:460] Webserver started at http://127.16.157.126:40267/ using document root <none> and password file <none>
I20260928 06:34:13.032486 27257 fs_manager.cc:359] Metadata directory not provided
I20260928 06:34:13.032567 27257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:34:13.032737 27257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:34:13.034493 27257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/minicluster-data/master-0/data/instance:
uuid: "0c01d6ab09364674936dad1425a28ebc"
format_stamp: "Formatted at 2026-09-28 06:34:13 on dist-test-slave-rrj5"
server_key: "e4205bc75c6254bd866c8d2f8804bc82"
server_key_iv: "44889d2838aa51385866a39e3ae8cbf0"
server_key_version: "encryptionkey@0"
I20260928 06:34:13.034965 27257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/minicluster-data/master-0/wal/instance:
uuid: "0c01d6ab09364674936dad1425a28ebc"
format_stamp: "Formatted at 2026-09-28 06:34:13 on dist-test-slave-rrj5"
server_key: "e4205bc75c6254bd866c8d2f8804bc82"
server_key_iv: "44889d2838aa51385866a39e3ae8cbf0"
server_key_version: "encryptionkey@0"
I20260928 06:34:13.036505 27257 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:34:13.037654 27272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:13.037851 27257 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:34:13.037961 27257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/minicluster-data/master-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/minicluster-data/master-0/wal
uuid: "0c01d6ab09364674936dad1425a28ebc"
format_stamp: "Formatted at 2026-09-28 06:34:13 on dist-test-slave-rrj5"
server_key: "e4205bc75c6254bd866c8d2f8804bc82"
server_key_iv: "44889d2838aa51385866a39e3ae8cbf0"
server_key_version: "encryptionkey@0"
I20260928 06:34:13.038056 27257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:34:13.054174 27257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:34:13.056804 27257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:34:13.056984 27257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:34:13.062865 27257 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.126:46459
I20260928 06:34:13.062953 27324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.126:46459 every 8 connection(s)
I20260928 06:34:13.063349 27257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
I20260928 06:34:13.064349 27325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:34:13.065146 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 27257
I20260928 06:34:13.065243 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/minicluster-data/master-0/wal/instance
I20260928 06:34:13.065440 17013 external_mini_cluster.cc:1428] Setting key ce0a71ed76487e97ac46a705a22e96a8
I20260928 06:34:13.068004 27325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c01d6ab09364674936dad1425a28ebc: Bootstrap starting.
I20260928 06:34:13.068715 27325 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c01d6ab09364674936dad1425a28ebc: Neither blocks nor log segments found. Creating new log.
I20260928 06:34:13.069074 27325 log.cc:824] T 00000000000000000000000000000000 P 0c01d6ab09364674936dad1425a28ebc: Log is configured to *not* fsync() on all Append() calls
I20260928 06:34:13.069890 27325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c01d6ab09364674936dad1425a28ebc: No bootstrap required, opened a new log
I20260928 06:34:13.072546 27325 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c01d6ab09364674936dad1425a28ebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c01d6ab09364674936dad1425a28ebc" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 46459 } }
I20260928 06:34:13.072744 27325 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c01d6ab09364674936dad1425a28ebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:34:13.072795 27325 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c01d6ab09364674936dad1425a28ebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c01d6ab09364674936dad1425a28ebc, State: Initialized, Role: FOLLOWER
I20260928 06:34:13.072943 27325 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c01d6ab09364674936dad1425a28ebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c01d6ab09364674936dad1425a28ebc" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 46459 } }
I20260928 06:34:13.073057 27325 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c01d6ab09364674936dad1425a28ebc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:34:13.073099 27325 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c01d6ab09364674936dad1425a28ebc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:34:13.073159 27325 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c01d6ab09364674936dad1425a28ebc [term 0 FOLLOWER]: Advancing to term 1
Sep 28 06:34:13 dist-test-slave-rrj5 krb5kdc[27244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:34:13.074014 27325 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c01d6ab09364674936dad1425a28ebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c01d6ab09364674936dad1425a28ebc" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 46459 } }
I20260928 06:34:13.074146 27325 leader_election.cc:302] T 00000000000000000000000000000000 P 0c01d6ab09364674936dad1425a28ebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c01d6ab09364674936dad1425a28ebc; no voters:
I20260928 06:34:13.074339 27325 leader_election.cc:288] T 00000000000000000000000000000000 P 0c01d6ab09364674936dad1425a28ebc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:34:13.074376 27330 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c01d6ab09364674936dad1425a28ebc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:34:13.074496 27330 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c01d6ab09364674936dad1425a28ebc [term 1 LEADER]: Becoming Leader. State: Replica: 0c01d6ab09364674936dad1425a28ebc, State: Running, Role: LEADER
I20260928 06:34:13.074581 27330 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c01d6ab09364674936dad1425a28ebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c01d6ab09364674936dad1425a28ebc" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 46459 } }
I20260928 06:34:13.074627 27325 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c01d6ab09364674936dad1425a28ebc [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:34:13.074913 27332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c01d6ab09364674936dad1425a28ebc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c01d6ab09364674936dad1425a28ebc. Latest consensus state: current_term: 1 leader_uuid: "0c01d6ab09364674936dad1425a28ebc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c01d6ab09364674936dad1425a28ebc" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 46459 } } }
I20260928 06:34:13.075049 27332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c01d6ab09364674936dad1425a28ebc [sys.catalog]: This master's current role is: LEADER
I20260928 06:34:13.074913 27331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c01d6ab09364674936dad1425a28ebc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c01d6ab09364674936dad1425a28ebc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c01d6ab09364674936dad1425a28ebc" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 46459 } } }
I20260928 06:34:13.075169 27331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c01d6ab09364674936dad1425a28ebc [sys.catalog]: This master's current role is: LEADER
I20260928 06:34:13.076012 27334 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:34:13.076570 27334 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:34:13.078562 27334 catalog_manager.cc:1347] Generated new cluster ID: 08b877e6a33642c5bffdb79e35c4cca9
I20260928 06:34:13.078617 27334 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:34:13.092845 27334 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:34:13.093809 27334 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:34:13.094274 27328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:13.066060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58316 (local address 127.16.157.126:46459)
0928 06:34:13.066240 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:13.066252 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:13.066294 (+ 42us) server_negotiation.cc:407] Connection header received
0928 06:34:13.066934 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:13.066941 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:13.066996 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:13.067158 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:13.067842 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:13.069235 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:13.070238 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:13.070708 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:13.070853 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:13.074051 (+ 3198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:34:13.074055 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:34:13.074055 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:34:13.074072 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:34:13.091120 (+ 17048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:34:13.091695 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:34:13.091699 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:34:13.091700 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:34:13.091840 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:34:13.092146 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:34:13.092147 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:34:13.092148 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:34:13.092675 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:34:13.093483 (+ 808us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:13.093609 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 06:34:13.093830 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":122,"thread_start_us":78,"threads_started":1}
I20260928 06:34:13.106542 27334 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c01d6ab09364674936dad1425a28ebc: Generated new TSK 0
I20260928 06:34:13.106762 27334 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.157.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab.
Entry for principal kudu/127.16.157.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab.
I20260928 06:34:13.141362 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.65:0
--local_ip_for_outbound_sockets=127.16.157.65
--webserver_interface=127.16.157.65
--webserver_port=0
--tserver_master_addrs=127.16.157.126:46459
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab
--principal=kudu/127.16.157.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:41133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:34:13.259109 27352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:34:13.259366 27352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:34:13.259399 27352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:34:13.261062 27352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:34:13.261116 27352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:34:13.261154 27352 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.65
I20260928 06:34:13.263161 27352 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:41133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab
--principal=kudu/127.16.157.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.157.65
--webserver_port=0
--tserver_master_addrs=127.16.157.126:46459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27352
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.65
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:34:13.263407 27352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:34:13.263644 27352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:34:13.269321 27360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:34:13.269321 27357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:34:13.269351 27358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:34:13.269677 27352 server_base.cc:1034] running on GCE node
I20260928 06:34:13.269872 27352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:34:13.270165 27352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:34:13.271328 27352 hybrid_clock.cc:648] HybridClock initialized: now 1790577253271305 us; error 44 us; skew 500 ppm
Sep 28 06:34:13 dist-test-slave-rrj5 krb5kdc[27244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577253, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:34:13.273901 27352 init.cc:377] Logged in from keytab as kudu/127.16.157.65@KRBTEST.COM (short username kudu)
I20260928 06:34:13.274375 27352 webserver.cc:460] Webserver started at http://127.16.157.65:39797/ using document root <none> and password file <none>
I20260928 06:34:13.274657 27352 fs_manager.cc:359] Metadata directory not provided
I20260928 06:34:13.274705 27352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:34:13.274820 27352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:34:13.276082 27352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/minicluster-data/ts-0/data/instance:
uuid: "dd4d3078692b43fe83340dfb79547f33"
format_stamp: "Formatted at 2026-09-28 06:34:13 on dist-test-slave-rrj5"
server_key: "399c537cf6149d504b1edfc366a8ebb9"
server_key_iv: "06763893045932dd38a3ab48c8337882"
server_key_version: "encryptionkey@0"
I20260928 06:34:13.276444 27352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/minicluster-data/ts-0/wal/instance:
uuid: "dd4d3078692b43fe83340dfb79547f33"
format_stamp: "Formatted at 2026-09-28 06:34:13 on dist-test-slave-rrj5"
server_key: "399c537cf6149d504b1edfc366a8ebb9"
server_key_iv: "06763893045932dd38a3ab48c8337882"
server_key_version: "encryptionkey@0"
I20260928 06:34:13.278132 27352 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:34:13.279120 27367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:13.279323 27352 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:34:13.279417 27352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/minicluster-data/ts-0/wal
uuid: "dd4d3078692b43fe83340dfb79547f33"
format_stamp: "Formatted at 2026-09-28 06:34:13 on dist-test-slave-rrj5"
server_key: "399c537cf6149d504b1edfc366a8ebb9"
server_key_iv: "06763893045932dd38a3ab48c8337882"
server_key_version: "encryptionkey@0"
I20260928 06:34:13.279502 27352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:34:13.299563 27352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:34:13.302189 27352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:34:13.302342 27352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:34:13.302559 27352 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:34:13.302872 27352 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:34:13.302924 27352 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:13.302966 27352 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:34:13.302994 27352 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:13.310043 27352 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.65:42721
I20260928 06:34:13.310123 27480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.65:42721 every 8 connection(s)
I20260928 06:34:13.310417 27352 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
Sep 28 06:34:13 dist-test-slave-rrj5 krb5kdc[27244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577253, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.65@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:34:13.317847 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 27352
I20260928 06:34:13.317965 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/minicluster-data/ts-0/wal/instance
I20260928 06:34:13.318162 17013 external_mini_cluster.cc:1428] Setting key 13b67956dc3eb77a6134f5e94c82c193
I20260928 06:34:13.321317 27328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:13.311440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:37913 (local address 127.16.157.126:46459)
0928 06:34:13.311489 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:13.311491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:13.311502 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:34:13.312003 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:13.312004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:13.312015 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:13.312088 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:13.313253 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:13.314147 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:13.315178 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:13.315549 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:13.315636 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:13.318032 (+ 2396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:34:13.318033 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:34:13.318034 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:34:13.318042 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:34:13.319594 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:34:13.320024 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:34:13.320025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:34:13.320026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:34:13.320114 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:34:13.320477 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:34:13.320478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:34:13.320478 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:34:13.320643 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:34:13.320727 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:13.321104 (+ 377us) server_negotiation.cc:299] Negotiation successful
0928 06:34:13.321205 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 06:34:13.321497 27483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:13.311151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:46459 (local address 127.16.157.65:37913)
0928 06:34:13.311329 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:13.311397 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:13.311900 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:13.312165 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:13.312169 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:13.312960 (+ 791us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:34:13.313152 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:13.313157 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:13.314264 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:13.314265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:13.315107 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:13.315109 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:13.315648 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:13.315649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:13.315760 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:13.316292 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:34:13.316299 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:34:13.317876 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:34:13.319713 (+ 1837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:34:13.319721 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:34:13.319723 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:34:13.319944 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:34:13.320233 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:34:13.320234 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:34:13.320234 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:34:13.320385 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:34:13.320728 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:34:13.320735 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:34:13.320925 (+ 190us) client_negotiation.cc:769] Sending connection context
0928 06:34:13.321053 (+ 128us) client_negotiation.cc:240] Negotiation successful
0928 06:34:13.321224 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":78,"threads_started":1}
I20260928 06:34:13.322322 27481 heartbeater.cc:347] Connected to a master server at 127.16.157.126:46459
I20260928 06:34:13.322423 27481 heartbeater.cc:464] Registering TS with master...
I20260928 06:34:13.322613 27481 heartbeater.cc:511] Master 127.16.157.126:46459 requested a full tablet report, sending...
I20260928 06:34:13.323262 27289 ts_manager.cc:194] Registered new tserver with Master: dd4d3078692b43fe83340dfb79547f33 (127.16.157.65:42721)
I20260928 06:34:13.324162 27289 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.157.65@KRBTEST.COM'} at 127.16.157.65:37913
WARNING: no policy specified for kudu/127.16.157.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab.
Entry for principal kudu/127.16.157.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab.
I20260928 06:34:13.348335 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.66:0
--local_ip_for_outbound_sockets=127.16.157.66
--webserver_interface=127.16.157.66
--webserver_port=0
--tserver_master_addrs=127.16.157.126:46459
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab
--principal=kudu/127.16.157.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:41133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:34:13.442179 27486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:34:13.442417 27486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:34:13.442461 27486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:34:13.444443 27486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:34:13.444507 27486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:34:13.444569 27486 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.66
I20260928 06:34:13.447073 27486 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:41133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab
--principal=kudu/127.16.157.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.157.66
--webserver_port=0
--tserver_master_addrs=127.16.157.126:46459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27486
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.66
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:34:13.447326 27486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:34:13.447584 27486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:34:13.453203 27492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:34:13.453230 27494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:34:13.453352 27486 server_base.cc:1034] running on GCE node
W20260928 06:34:13.453471 27491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:34:13.453723 27486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:34:13.454030 27486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:34:13.455196 27486 hybrid_clock.cc:648] HybridClock initialized: now 1790577253455187 us; error 34 us; skew 500 ppm
Sep 28 06:34:13 dist-test-slave-rrj5 krb5kdc[27244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577253, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:34:13.458308 27486 init.cc:377] Logged in from keytab as kudu/127.16.157.66@KRBTEST.COM (short username kudu)
I20260928 06:34:13.458853 27486 webserver.cc:460] Webserver started at http://127.16.157.66:46779/ using document root <none> and password file <none>
I20260928 06:34:13.459100 27486 fs_manager.cc:359] Metadata directory not provided
I20260928 06:34:13.459170 27486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:34:13.459308 27486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:34:13.460757 27486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/minicluster-data/ts-1/data/instance:
uuid: "af6ca90f9bf440aab1d4d494162dbd1f"
format_stamp: "Formatted at 2026-09-28 06:34:13 on dist-test-slave-rrj5"
server_key: "2ea36ba251b964d7532581893c83d021"
server_key_iv: "294141aa92f4ac054fa87ebfb9da04fe"
server_key_version: "encryptionkey@0"
I20260928 06:34:13.461189 27486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/minicluster-data/ts-1/wal/instance:
uuid: "af6ca90f9bf440aab1d4d494162dbd1f"
format_stamp: "Formatted at 2026-09-28 06:34:13 on dist-test-slave-rrj5"
server_key: "2ea36ba251b964d7532581893c83d021"
server_key_iv: "294141aa92f4ac054fa87ebfb9da04fe"
server_key_version: "encryptionkey@0"
I20260928 06:34:13.462949 27486 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260928 06:34:13.463766 27501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:13.463961 27486 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:34:13.464073 27486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/minicluster-data/ts-1/wal
uuid: "af6ca90f9bf440aab1d4d494162dbd1f"
format_stamp: "Formatted at 2026-09-28 06:34:13 on dist-test-slave-rrj5"
server_key: "2ea36ba251b964d7532581893c83d021"
server_key_iv: "294141aa92f4ac054fa87ebfb9da04fe"
server_key_version: "encryptionkey@0"
I20260928 06:34:13.464170 27486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:34:13.473939 27486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:34:13.476457 27486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:34:13.476619 27486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:34:13.476846 27486 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:34:13.477169 27486 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:34:13.477229 27486 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:13.477269 27486 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:34:13.477294 27486 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:13.483505 27486 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.66:33681
I20260928 06:34:13.483557 27615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.66:33681 every 8 connection(s)
I20260928 06:34:13.483916 27486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
Sep 28 06:34:13 dist-test-slave-rrj5 krb5kdc[27244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577253, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.66@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:34:13.494071 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 27486
I20260928 06:34:13.494174 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/minicluster-data/ts-1/wal/instance
I20260928 06:34:13.494371 17013 external_mini_cluster.cc:1428] Setting key 048941887b934efd790faba316a9fa0b
I20260928 06:34:13.494377 27328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:13.484638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.66:43065 (local address 127.16.157.126:46459)
0928 06:34:13.484722 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:13.484725 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:13.484944 (+ 219us) server_negotiation.cc:407] Connection header received
0928 06:34:13.485476 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:13.485478 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:13.485492 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:13.485592 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:13.486680 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:13.487556 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:13.488665 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:13.489017 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:13.489108 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:13.491362 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:34:13.491363 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:34:13.491364 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:34:13.491372 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:34:13.492822 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:34:13.493303 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:34:13.493305 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:34:13.493305 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:34:13.493366 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:34:13.493676 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:34:13.493677 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:34:13.493677 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:34:13.493795 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:34:13.493877 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:13.494205 (+ 328us) server_negotiation.cc:299] Negotiation successful
0928 06:34:13.494287 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 06:34:13.494546 27618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:13.484599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:46459 (local address 127.16.157.66:43065)
0928 06:34:13.484795 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:13.484846 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:13.485372 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:13.485695 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:13.485699 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:13.486338 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:34:13.486599 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:13.486604 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:13.487664 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:13.487664 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:13.488580 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:13.488582 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:13.489117 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:13.489117 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:13.489220 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:13.489804 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:34:13.489812 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:34:13.491267 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:34:13.492946 (+ 1679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:34:13.492953 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:34:13.492955 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:34:13.493183 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:34:13.493435 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:34:13.493436 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:34:13.493436 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:34:13.493548 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:34:13.493889 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:34:13.493896 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:34:13.494045 (+ 149us) client_negotiation.cc:769] Sending connection context
0928 06:34:13.494159 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 06:34:13.494315 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":112,"threads_started":1}
I20260928 06:34:13.495620 27616 heartbeater.cc:347] Connected to a master server at 127.16.157.126:46459
I20260928 06:34:13.495735 27616 heartbeater.cc:464] Registering TS with master...
I20260928 06:34:13.495919 27616 heartbeater.cc:511] Master 127.16.157.126:46459 requested a full tablet report, sending...
I20260928 06:34:13.496302 27289 ts_manager.cc:194] Registered new tserver with Master: af6ca90f9bf440aab1d4d494162dbd1f (127.16.157.66:33681)
I20260928 06:34:13.496799 27289 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.157.66@KRBTEST.COM'} at 127.16.157.66:43065
WARNING: no policy specified for kudu/127.16.157.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab.
Entry for principal kudu/127.16.157.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab.
I20260928 06:34:13.521232 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.67:0
--local_ip_for_outbound_sockets=127.16.157.67
--webserver_interface=127.16.157.67
--webserver_port=0
--tserver_master_addrs=127.16.157.126:46459
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab
--principal=kudu/127.16.157.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:41133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:34:13.614351 27621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:34:13.614615 27621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:34:13.614672 27621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:34:13.616044 27621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:34:13.616088 27621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:34:13.616115 27621 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.67
I20260928 06:34:13.617708 27621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:41133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab
--principal=kudu/127.16.157.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.157.67
--webserver_port=0
--tserver_master_addrs=127.16.157.126:46459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.67
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:34:13.617956 27621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:34:13.618183 27621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:34:13.623622 27626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:34:13.623761 27629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:34:13.623643 27627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:34:13.624017 27621 server_base.cc:1034] running on GCE node
I20260928 06:34:13.624244 27621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:34:13.624485 27621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:34:13.625656 27621 hybrid_clock.cc:648] HybridClock initialized: now 1790577253625642 us; error 32 us; skew 500 ppm
Sep 28 06:34:13 dist-test-slave-rrj5 krb5kdc[27244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577253, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:34:13.628587 27621 init.cc:377] Logged in from keytab as kudu/127.16.157.67@KRBTEST.COM (short username kudu)
I20260928 06:34:13.628983 27621 webserver.cc:460] Webserver started at http://127.16.157.67:42729/ using document root <none> and password file <none>
I20260928 06:34:13.629204 27621 fs_manager.cc:359] Metadata directory not provided
I20260928 06:34:13.629268 27621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:34:13.629395 27621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:34:13.630472 27621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/minicluster-data/ts-2/data/instance:
uuid: "3e73989113cb44ee949db46590ad401c"
format_stamp: "Formatted at 2026-09-28 06:34:13 on dist-test-slave-rrj5"
server_key: "560e32250df77fea16af6014622ba44c"
server_key_iv: "1cacd42776edf7f410973eaf9475f9a8"
server_key_version: "encryptionkey@0"
I20260928 06:34:13.630831 27621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/minicluster-data/ts-2/wal/instance:
uuid: "3e73989113cb44ee949db46590ad401c"
format_stamp: "Formatted at 2026-09-28 06:34:13 on dist-test-slave-rrj5"
server_key: "560e32250df77fea16af6014622ba44c"
server_key_iv: "1cacd42776edf7f410973eaf9475f9a8"
server_key_version: "encryptionkey@0"
I20260928 06:34:13.632151 27621 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:34:13.633093 27636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:13.633277 27621 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:34:13.633368 27621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/minicluster-data/ts-2/wal
uuid: "3e73989113cb44ee949db46590ad401c"
format_stamp: "Formatted at 2026-09-28 06:34:13 on dist-test-slave-rrj5"
server_key: "560e32250df77fea16af6014622ba44c"
server_key_iv: "1cacd42776edf7f410973eaf9475f9a8"
server_key_version: "encryptionkey@0"
I20260928 06:34:13.633455 27621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:34:13.647279 27621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:34:13.649732 27621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:34:13.649889 27621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:34:13.650128 27621 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:34:13.650420 27621 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:34:13.650472 27621 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:13.650512 27621 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:34:13.650565 27621 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:13.656984 27621 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.67:40217
I20260928 06:34:13.657050 27750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.67:40217 every 8 connection(s)
I20260928 06:34:13.657374 27621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
Sep 28 06:34:13 dist-test-slave-rrj5 krb5kdc[27244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577253, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.67@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:34:13.666410 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 27621
I20260928 06:34:13.666517 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/minicluster-data/ts-2/wal/instance
I20260928 06:34:13.666728 17013 external_mini_cluster.cc:1428] Setting key 7c24180f27dd55c03c854a3e48018e66
I20260928 06:34:13.667825 27328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:13.658175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:56733 (local address 127.16.157.126:46459)
0928 06:34:13.658218 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:13.658221 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:13.658452 (+ 231us) server_negotiation.cc:407] Connection header received
0928 06:34:13.658946 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:13.658948 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:13.658962 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:13.659035 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:13.660347 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:13.660878 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:13.661939 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:13.662184 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:13.662246 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:13.664364 (+ 2118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:34:13.664366 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:34:13.664367 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:34:13.664377 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:34:13.666069 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:34:13.666641 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:34:13.666642 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:34:13.666643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:34:13.666714 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:34:13.667053 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:34:13.667054 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:34:13.667055 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:34:13.667212 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:34:13.667280 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:13.667625 (+ 345us) server_negotiation.cc:299] Negotiation successful
0928 06:34:13.667728 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:34:13.668054 27753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:13.658168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:46459 (local address 127.16.157.67:56733)
0928 06:34:13.658342 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:13.658376 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:13.658837 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:13.659139 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:13.659143 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:13.659952 (+ 809us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:34:13.660246 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:13.660254 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:13.660981 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:13.660982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:13.661851 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:13.661853 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:13.662285 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:13.662286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:13.662435 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:13.662950 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:34:13.662966 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:34:13.664263 (+ 1297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:34:13.666181 (+ 1918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:34:13.666188 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:34:13.666190 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:34:13.666553 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:34:13.666804 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:34:13.666805 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:34:13.666805 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:34:13.666957 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:34:13.667335 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:34:13.667342 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:34:13.667479 (+ 137us) client_negotiation.cc:769] Sending connection context
0928 06:34:13.667600 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 06:34:13.667775 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":102,"threads_started":1}
I20260928 06:34:13.668653 27751 heartbeater.cc:347] Connected to a master server at 127.16.157.126:46459
I20260928 06:34:13.668763 27751 heartbeater.cc:464] Registering TS with master...
I20260928 06:34:13.668949 27751 heartbeater.cc:511] Master 127.16.157.126:46459 requested a full tablet report, sending...
I20260928 06:34:13.669400 27289 ts_manager.cc:194] Registered new tserver with Master: 3e73989113cb44ee949db46590ad401c (127.16.157.67:40217)
I20260928 06:34:13.669829 27289 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.157.67@KRBTEST.COM'} at 127.16.157.67:56733
I20260928 06:34:13.678865 17013 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:34:13.686726 27328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:13.680284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58326 (local address 127.16.157.126:46459)
0928 06:34:13.680323 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:13.680325 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:13.680465 (+ 140us) server_negotiation.cc:407] Connection header received
0928 06:34:13.680547 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:13.680548 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:13.680557 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:13.680612 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:13.681185 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:13.681846 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:13.682597 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:13.682799 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:13.682873 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:13.684122 (+ 1249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:34:13.684124 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:34:13.684124 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:34:13.684132 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:34:13.685537 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:34:13.685919 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:34:13.685921 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:34:13.685921 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:34:13.685983 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:34:13.686185 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:34:13.686186 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:34:13.686186 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:34:13.686336 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:34:13.686441 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:13.686548 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 06:34:13.686627 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:34:13.688457 27289 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58326:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:34:13.690080 27289 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:34:13.697036 27762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:13.692738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.65:42721 (local address 127.0.0.1:53532)
0928 06:34:13.693289 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:13.693299 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:13.693340 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:13.693892 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:13.693893 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:13.693895 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:34:13.693947 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:13.693948 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:13.695257 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:13.695258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:13.696012 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:13.696014 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:13.696677 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:13.696677 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:13.696753 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:13.696766 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 06:34:13.696893 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 06:34:13.696953 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":79,"threads_started":1}
I20260928 06:34:13.697039 27767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:13.692953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53532 (local address 127.16.157.65:42721)
0928 06:34:13.693715 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:13.693719 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:13.693729 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:34:13.693770 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:13.693772 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:13.693787 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:13.694023 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:34:13.694126 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:13.695150 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:13.696109 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:13.696591 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:13.696676 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:13.696736 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:13.696914 (+ 178us) server_negotiation.cc:299] Negotiation successful
0928 06:34:13.696960 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":721,"thread_start_us":74,"threads_started":1}
I20260928 06:34:13.697315 27763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:13.692526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.67:40217 (local address 127.0.0.1:52564)
0928 06:34:13.692692 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:13.692734 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:13.692799 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:13.693311 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:13.693312 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:13.693314 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:34:13.693370 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:13.693372 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:13.694657 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:13.694657 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:13.696054 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:13.696056 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:13.697066 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:13.697066 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:13.697135 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:13.697139 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:34:13.697193 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 06:34:13.697239 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":58,"thread_start_us":60,"threads_started":1}
I20260928 06:34:13.697481 27766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:13.692792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58524 (local address 127.16.157.66:33681)
0928 06:34:13.693039 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:13.693044 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:13.693054 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:34:13.693094 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:13.693096 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:13.693114 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:13.693195 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:34:13.693353 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:13.694529 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:13.696233 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:13.696891 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:13.696968 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:13.697024 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:13.697352 (+ 328us) server_negotiation.cc:299] Negotiation successful
0928 06:34:13.697403 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":77,"threads_started":1}
I20260928 06:34:13.697566 27764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:13.692616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52564 (local address 127.16.157.67:40217)
0928 06:34:13.692992 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:13.692997 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:13.693008 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:34:13.693053 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:13.693055 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:13.693149 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:13.693229 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:34:13.693452 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:13.694546 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:13.696151 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:13.696891 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:13.696980 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:13.697024 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:13.697430 (+ 406us) server_negotiation.cc:299] Negotiation successful
0928 06:34:13.697471 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":76,"threads_started":1}
I20260928 06:34:13.697666 27765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:13.692550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.66:33681 (local address 127.0.0.1:58524)
0928 06:34:13.692862 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:13.692873 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:13.692925 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:13.693206 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:13.693210 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:13.693213 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:34:13.693279 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:13.693281 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:13.694656 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:13.694657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:13.696151 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:13.696152 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:13.697064 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:13.697065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:13.697142 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:13.697144 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:34:13.697264 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 06:34:13.697297 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":3,"thread_start_us":86,"threads_started":1}
I20260928 06:34:13.697935 27415 tablet_service.cc:1467] Processing CreateTablet for tablet a3d0d1ef1cdb409b85440c892e493e25 (DEFAULT_TABLE table=test-table [id=f80ea976cd8042c99e3285e8d7c7af4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:34:13.698289 27415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3d0d1ef1cdb409b85440c892e493e25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:34:13.698534 27685 tablet_service.cc:1467] Processing CreateTablet for tablet a3d0d1ef1cdb409b85440c892e493e25 (DEFAULT_TABLE table=test-table [id=f80ea976cd8042c99e3285e8d7c7af4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:34:13.698910 27685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3d0d1ef1cdb409b85440c892e493e25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:34:13.701457 27768 tablet_bootstrap.cc:492] T a3d0d1ef1cdb409b85440c892e493e25 P dd4d3078692b43fe83340dfb79547f33: Bootstrap starting.
I20260928 06:34:13.702104 27769 tablet_bootstrap.cc:492] T a3d0d1ef1cdb409b85440c892e493e25 P 3e73989113cb44ee949db46590ad401c: Bootstrap starting.
I20260928 06:34:13.702159 27768 tablet_bootstrap.cc:654] T a3d0d1ef1cdb409b85440c892e493e25 P dd4d3078692b43fe83340dfb79547f33: Neither blocks nor log segments found. Creating new log.
I20260928 06:34:13.702665 27768 log.cc:824] T a3d0d1ef1cdb409b85440c892e493e25 P dd4d3078692b43fe83340dfb79547f33: Log is configured to *not* fsync() on all Append() calls
I20260928 06:34:13.702853 27769 tablet_bootstrap.cc:654] T a3d0d1ef1cdb409b85440c892e493e25 P 3e73989113cb44ee949db46590ad401c: Neither blocks nor log segments found. Creating new log.
I20260928 06:34:13.703387 27769 log.cc:824] T a3d0d1ef1cdb409b85440c892e493e25 P 3e73989113cb44ee949db46590ad401c: Log is configured to *not* fsync() on all Append() calls
I20260928 06:34:13.703485 27768 tablet_bootstrap.cc:492] T a3d0d1ef1cdb409b85440c892e493e25 P dd4d3078692b43fe83340dfb79547f33: No bootstrap required, opened a new log
I20260928 06:34:13.703541 27768 ts_tablet_manager.cc:1397] T a3d0d1ef1cdb409b85440c892e493e25 P dd4d3078692b43fe83340dfb79547f33: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:34:13.703639 27550 tablet_service.cc:1467] Processing CreateTablet for tablet a3d0d1ef1cdb409b85440c892e493e25 (DEFAULT_TABLE table=test-table [id=f80ea976cd8042c99e3285e8d7c7af4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:34:13.703929 27550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3d0d1ef1cdb409b85440c892e493e25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:34:13.704048 27769 tablet_bootstrap.cc:492] T a3d0d1ef1cdb409b85440c892e493e25 P 3e73989113cb44ee949db46590ad401c: No bootstrap required, opened a new log
I20260928 06:34:13.704090 27769 ts_tablet_manager.cc:1397] T a3d0d1ef1cdb409b85440c892e493e25 P 3e73989113cb44ee949db46590ad401c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:34:13.705410 27768 raft_consensus.cc:348] T a3d0d1ef1cdb409b85440c892e493e25 P dd4d3078692b43fe83340dfb79547f33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e73989113cb44ee949db46590ad401c" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 40217 } } peers { permanent_uuid: "dd4d3078692b43fe83340dfb79547f33" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 42721 } } peers { permanent_uuid: "af6ca90f9bf440aab1d4d494162dbd1f" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 33681 } }
I20260928 06:34:13.705461 27769 raft_consensus.cc:348] T a3d0d1ef1cdb409b85440c892e493e25 P 3e73989113cb44ee949db46590ad401c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e73989113cb44ee949db46590ad401c" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 40217 } } peers { permanent_uuid: "dd4d3078692b43fe83340dfb79547f33" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 42721 } } peers { permanent_uuid: "af6ca90f9bf440aab1d4d494162dbd1f" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 33681 } }
I20260928 06:34:13.705606 27768 raft_consensus.cc:374] T a3d0d1ef1cdb409b85440c892e493e25 P dd4d3078692b43fe83340dfb79547f33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:34:13.705638 27769 raft_consensus.cc:374] T a3d0d1ef1cdb409b85440c892e493e25 P 3e73989113cb44ee949db46590ad401c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:34:13.705703 27768 raft_consensus.cc:729] T a3d0d1ef1cdb409b85440c892e493e25 P dd4d3078692b43fe83340dfb79547f33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd4d3078692b43fe83340dfb79547f33, State: Initialized, Role: FOLLOWER
I20260928 06:34:13.705703 27769 raft_consensus.cc:729] T a3d0d1ef1cdb409b85440c892e493e25 P 3e73989113cb44ee949db46590ad401c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e73989113cb44ee949db46590ad401c, State: Initialized, Role: FOLLOWER
I20260928 06:34:13.705811 27768 consensus_queue.cc:260] T a3d0d1ef1cdb409b85440c892e493e25 P dd4d3078692b43fe83340dfb79547f33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e73989113cb44ee949db46590ad401c" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 40217 } } peers { permanent_uuid: "dd4d3078692b43fe83340dfb79547f33" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 42721 } } peers { permanent_uuid: "af6ca90f9bf440aab1d4d494162dbd1f" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 33681 } }
I20260928 06:34:13.705840 27769 consensus_queue.cc:260] T a3d0d1ef1cdb409b85440c892e493e25 P 3e73989113cb44ee949db46590ad401c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e73989113cb44ee949db46590ad401c" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 40217 } } peers { permanent_uuid: "dd4d3078692b43fe83340dfb79547f33" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 42721 } } peers { permanent_uuid: "af6ca90f9bf440aab1d4d494162dbd1f" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 33681 } }
I20260928 06:34:13.706069 27768 ts_tablet_manager.cc:1428] T a3d0d1ef1cdb409b85440c892e493e25 P dd4d3078692b43fe83340dfb79547f33: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 06:34:13.706148 27769 ts_tablet_manager.cc:1428] T a3d0d1ef1cdb409b85440c892e493e25 P 3e73989113cb44ee949db46590ad401c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:34:13.706195 27481 heartbeater.cc:503] Master 127.16.157.126:46459 was elected leader, sending a full tablet report...
I20260928 06:34:13.706274 27751 heartbeater.cc:503] Master 127.16.157.126:46459 was elected leader, sending a full tablet report...
I20260928 06:34:13.707458 27774 tablet_bootstrap.cc:492] T a3d0d1ef1cdb409b85440c892e493e25 P af6ca90f9bf440aab1d4d494162dbd1f: Bootstrap starting.
I20260928 06:34:13.708277 27774 tablet_bootstrap.cc:654] T a3d0d1ef1cdb409b85440c892e493e25 P af6ca90f9bf440aab1d4d494162dbd1f: Neither blocks nor log segments found. Creating new log.
I20260928 06:34:13.708774 27774 log.cc:824] T a3d0d1ef1cdb409b85440c892e493e25 P af6ca90f9bf440aab1d4d494162dbd1f: Log is configured to *not* fsync() on all Append() calls
I20260928 06:34:13.709532 27774 tablet_bootstrap.cc:492] T a3d0d1ef1cdb409b85440c892e493e25 P af6ca90f9bf440aab1d4d494162dbd1f: No bootstrap required, opened a new log
I20260928 06:34:13.709625 27774 ts_tablet_manager.cc:1397] T a3d0d1ef1cdb409b85440c892e493e25 P af6ca90f9bf440aab1d4d494162dbd1f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:34:13.710844 27774 raft_consensus.cc:348] T a3d0d1ef1cdb409b85440c892e493e25 P af6ca90f9bf440aab1d4d494162dbd1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e73989113cb44ee949db46590ad401c" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 40217 } } peers { permanent_uuid: "dd4d3078692b43fe83340dfb79547f33" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 42721 } } peers { permanent_uuid: "af6ca90f9bf440aab1d4d494162dbd1f" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 33681 } }
I20260928 06:34:13.710960 27774 raft_consensus.cc:374] T a3d0d1ef1cdb409b85440c892e493e25 P af6ca90f9bf440aab1d4d494162dbd1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:34:13.711004 27774 raft_consensus.cc:729] T a3d0d1ef1cdb409b85440c892e493e25 P af6ca90f9bf440aab1d4d494162dbd1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af6ca90f9bf440aab1d4d494162dbd1f, State: Initialized, Role: FOLLOWER
I20260928 06:34:13.711109 27774 consensus_queue.cc:260] T a3d0d1ef1cdb409b85440c892e493e25 P af6ca90f9bf440aab1d4d494162dbd1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e73989113cb44ee949db46590ad401c" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 40217 } } peers { permanent_uuid: "dd4d3078692b43fe83340dfb79547f33" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 42721 } } peers { permanent_uuid: "af6ca90f9bf440aab1d4d494162dbd1f" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 33681 } }
I20260928 06:34:13.711373 27774 ts_tablet_manager.cc:1428] T a3d0d1ef1cdb409b85440c892e493e25 P af6ca90f9bf440aab1d4d494162dbd1f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 06:34:13.711428 27616 heartbeater.cc:503] Master 127.16.157.126:46459 was elected leader, sending a full tablet report...
I20260928 06:34:13.724805 27772 raft_consensus.cc:482] T a3d0d1ef1cdb409b85440c892e493e25 P dd4d3078692b43fe83340dfb79547f33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:34:13.724887 27772 raft_consensus.cc:504] T a3d0d1ef1cdb409b85440c892e493e25 P dd4d3078692b43fe83340dfb79547f33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e73989113cb44ee949db46590ad401c" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 40217 } } peers { permanent_uuid: "dd4d3078692b43fe83340dfb79547f33" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 42721 } } peers { permanent_uuid: "af6ca90f9bf440aab1d4d494162dbd1f" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 33681 } }
I20260928 06:34:13.725241 27772 leader_election.cc:288] T a3d0d1ef1cdb409b85440c892e493e25 P dd4d3078692b43fe83340dfb79547f33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e73989113cb44ee949db46590ad401c (127.16.157.67:40217), af6ca90f9bf440aab1d4d494162dbd1f (127.16.157.66:33681)
I20260928 06:34:13.728472 27483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:13.725363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.67:40217 (local address 127.16.157.65:41983)
0928 06:34:13.725420 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:13.725449 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:13.725489 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:13.725845 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:13.725846 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:13.725848 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:34:13.725916 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:13.725918 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:13.726764 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:13.726765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:13.727544 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:13.727546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:13.728210 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:13.728211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:13.728291 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:13.728295 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:34:13.728348 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 06:34:13.728399 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":39}
I20260928 06:34:13.728753 27764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:13.725639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:41983 (local address 127.16.157.67:40217)
0928 06:34:13.725688 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:13.725690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:13.725699 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:34:13.725724 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:13.725725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:13.725735 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:13.725794 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:34:13.725976 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:13.726670 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:13.727681 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:13.728125 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:13.728215 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:13.728225 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:13.728650 (+ 425us) server_negotiation.cc:299] Negotiation successful
0928 06:34:13.728693 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 06:34:13.729146 27705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d0d1ef1cdb409b85440c892e493e25" candidate_uuid: "dd4d3078692b43fe83340dfb79547f33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e73989113cb44ee949db46590ad401c" is_pre_election: true
I20260928 06:34:13.729306 27766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:13.725639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:57141 (local address 127.16.157.66:33681)
0928 06:34:13.725681 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:13.725683 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:13.725690 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:34:13.725712 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:13.725713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:13.725721 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:13.725790 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:34:13.725929 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:13.726665 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:13.728299 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:13.728871 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:13.728986 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:13.728994 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:13.729206 (+ 212us) server_negotiation.cc:299] Negotiation successful
0928 06:34:13.729247 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:34:13.729312 27705 raft_consensus.cc:2454] T a3d0d1ef1cdb409b85440c892e493e25 P 3e73989113cb44ee949db46590ad401c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd4d3078692b43fe83340dfb79547f33 in term 0.
I20260928 06:34:13.729533 27777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:13.725378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.66:33681 (local address 127.16.157.65:57141)
0928 06:34:13.725559 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:13.725583 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:13.725636 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:13.725794 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:13.725795 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:13.725797 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:34:13.725862 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:13.725864 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:13.726756 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:13.726757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:13.728142 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:13.728145 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:13.729030 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:13.729032 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:13.729103 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:13.729106 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:34:13.729411 (+ 305us) client_negotiation.cc:240] Negotiation successful
0928 06:34:13.729452 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":1,"thread_start_us":115,"threads_started":1}
I20260928 06:34:13.729594 27370 leader_election.cc:302] T a3d0d1ef1cdb409b85440c892e493e25 P dd4d3078692b43fe83340dfb79547f33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e73989113cb44ee949db46590ad401c, dd4d3078692b43fe83340dfb79547f33; no voters:
I20260928 06:34:13.729737 27772 raft_consensus.cc:2781] T a3d0d1ef1cdb409b85440c892e493e25 P dd4d3078692b43fe83340dfb79547f33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:34:13.729795 27772 raft_consensus.cc:482] T a3d0d1ef1cdb409b85440c892e493e25 P dd4d3078692b43fe83340dfb79547f33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:34:13.729831 27772 raft_consensus.cc:3037] T a3d0d1ef1cdb409b85440c892e493e25 P dd4d3078692b43fe83340dfb79547f33 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:34:13.730168 27570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d0d1ef1cdb409b85440c892e493e25" candidate_uuid: "dd4d3078692b43fe83340dfb79547f33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af6ca90f9bf440aab1d4d494162dbd1f" is_pre_election: true
I20260928 06:34:13.730294 27570 raft_consensus.cc:2454] T a3d0d1ef1cdb409b85440c892e493e25 P af6ca90f9bf440aab1d4d494162dbd1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd4d3078692b43fe83340dfb79547f33 in term 0.
I20260928 06:34:13.730731 27772 raft_consensus.cc:504] T a3d0d1ef1cdb409b85440c892e493e25 P dd4d3078692b43fe83340dfb79547f33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e73989113cb44ee949db46590ad401c" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 40217 } } peers { permanent_uuid: "dd4d3078692b43fe83340dfb79547f33" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 42721 } } peers { permanent_uuid: "af6ca90f9bf440aab1d4d494162dbd1f" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 33681 } }
I20260928 06:34:13.730892 27772 leader_election.cc:288] T a3d0d1ef1cdb409b85440c892e493e25 P dd4d3078692b43fe83340dfb79547f33 [CANDIDATE]: Term 1 election: Requested vote from peers 3e73989113cb44ee949db46590ad401c (127.16.157.67:40217), af6ca90f9bf440aab1d4d494162dbd1f (127.16.157.66:33681)
I20260928 06:34:13.731014 27705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d0d1ef1cdb409b85440c892e493e25" candidate_uuid: "dd4d3078692b43fe83340dfb79547f33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e73989113cb44ee949db46590ad401c"
I20260928 06:34:13.731087 27705 raft_consensus.cc:3037] T a3d0d1ef1cdb409b85440c892e493e25 P 3e73989113cb44ee949db46590ad401c [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:34:13.731150 27570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d0d1ef1cdb409b85440c892e493e25" candidate_uuid: "dd4d3078692b43fe83340dfb79547f33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af6ca90f9bf440aab1d4d494162dbd1f"
I20260928 06:34:13.731235 27570 raft_consensus.cc:3037] T a3d0d1ef1cdb409b85440c892e493e25 P af6ca90f9bf440aab1d4d494162dbd1f [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:34:13.731837 27705 raft_consensus.cc:2454] T a3d0d1ef1cdb409b85440c892e493e25 P 3e73989113cb44ee949db46590ad401c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd4d3078692b43fe83340dfb79547f33 in term 1.
I20260928 06:34:13.731993 27370 leader_election.cc:302] T a3d0d1ef1cdb409b85440c892e493e25 P dd4d3078692b43fe83340dfb79547f33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e73989113cb44ee949db46590ad401c, dd4d3078692b43fe83340dfb79547f33; no voters:
I20260928 06:34:13.732082 27772 raft_consensus.cc:2781] T a3d0d1ef1cdb409b85440c892e493e25 P dd4d3078692b43fe83340dfb79547f33 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:34:13.732198 27772 raft_consensus.cc:686] T a3d0d1ef1cdb409b85440c892e493e25 P dd4d3078692b43fe83340dfb79547f33 [term 1 LEADER]: Becoming Leader. State: Replica: dd4d3078692b43fe83340dfb79547f33, State: Running, Role: LEADER
I20260928 06:34:13.732190 27570 raft_consensus.cc:2454] T a3d0d1ef1cdb409b85440c892e493e25 P af6ca90f9bf440aab1d4d494162dbd1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd4d3078692b43fe83340dfb79547f33 in term 1.
I20260928 06:34:13.732291 27772 consensus_queue.cc:237] T a3d0d1ef1cdb409b85440c892e493e25 P dd4d3078692b43fe83340dfb79547f33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e73989113cb44ee949db46590ad401c" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 40217 } } peers { permanent_uuid: "dd4d3078692b43fe83340dfb79547f33" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 42721 } } peers { permanent_uuid: "af6ca90f9bf440aab1d4d494162dbd1f" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 33681 } }
I20260928 06:34:13.733062 27289 catalog_manager.cc:5495] T a3d0d1ef1cdb409b85440c892e493e25 P dd4d3078692b43fe83340dfb79547f33 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd4d3078692b43fe83340dfb79547f33 (127.16.157.65). New cstate: current_term: 1 leader_uuid: "dd4d3078692b43fe83340dfb79547f33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e73989113cb44ee949db46590ad401c" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 40217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd4d3078692b43fe83340dfb79547f33" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 42721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af6ca90f9bf440aab1d4d494162dbd1f" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 33681 } health_report { overall_health: UNKNOWN } } }
W20260928 06:34:13.734431 27617 tablet.cc:2369] T a3d0d1ef1cdb409b85440c892e493e25 P af6ca90f9bf440aab1d4d494162dbd1f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:34:13.743744 27767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:13.741176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53538 (local address 127.16.157.65:42721)
0928 06:34:13.741240 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:13.741242 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:13.741262 (+ 20us) server_negotiation.cc:407] Connection header received
0928 06:34:13.741299 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:13.741299 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:13.741313 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:13.741385 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:34:13.741508 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:13.742018 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:13.742919 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:13.743108 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:13.743182 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:13.743378 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:34:13.743469 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:34:13.743589 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:13.743612 (+ 23us) server_negotiation.cc:299] Negotiation successful
0928 06:34:13.743632 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 06:34:13.745488 27705 raft_consensus.cc:1261] T a3d0d1ef1cdb409b85440c892e493e25 P 3e73989113cb44ee949db46590ad401c [term 1 FOLLOWER]: Refusing update from remote peer dd4d3078692b43fe83340dfb79547f33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:34:13.745543 27570 raft_consensus.cc:1261] T a3d0d1ef1cdb409b85440c892e493e25 P af6ca90f9bf440aab1d4d494162dbd1f [term 1 FOLLOWER]: Refusing update from remote peer dd4d3078692b43fe83340dfb79547f33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:34:13.745872 27772 consensus_queue.cc:1035] T a3d0d1ef1cdb409b85440c892e493e25 P dd4d3078692b43fe83340dfb79547f33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af6ca90f9bf440aab1d4d494162dbd1f" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 33681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:34:13.746018 27778 consensus_queue.cc:1035] T a3d0d1ef1cdb409b85440c892e493e25 P dd4d3078692b43fe83340dfb79547f33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e73989113cb44ee949db46590ad401c" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 40217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:34:13.748648 27782 mvcc.cc:205] Tried to move back new op lower bound from 7334204431338663936 to 7334204431287943168. Current Snapshot: MvccSnapshot[applied={T|T < 7334204431338663936}]
I20260928 06:34:13.749009 27783 mvcc.cc:205] Tried to move back new op lower bound from 7334204431338663936 to 7334204431287943168. Current Snapshot: MvccSnapshot[applied={T|T < 7334204431338663936}]
I20260928 06:34:13.750818 27289 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:34:13.750923 27289 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:34:13.751652 27289 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0c01d6ab09364674936dad1425a28ebc: Sending DeleteTablet for 3 replicas of tablet a3d0d1ef1cdb409b85440c892e493e25
I20260928 06:34:13.751965 27685 tablet_service.cc:1514] Processing DeleteTablet for tablet a3d0d1ef1cdb409b85440c892e493e25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:34:13 UTC) from {username='kudu', principal='kudu/127.16.157.126@KRBTEST.COM'} at 127.0.0.1:52564
I20260928 06:34:13.751983 27550 tablet_service.cc:1514] Processing DeleteTablet for tablet a3d0d1ef1cdb409b85440c892e493e25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:34:13 UTC) from {username='kudu', principal='kudu/127.16.157.126@KRBTEST.COM'} at 127.0.0.1:58524
I20260928 06:34:13.752005 27415 tablet_service.cc:1514] Processing DeleteTablet for tablet a3d0d1ef1cdb409b85440c892e493e25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:34:13 UTC) from {username='kudu', principal='kudu/127.16.157.126@KRBTEST.COM'} at 127.0.0.1:53532
I20260928 06:34:13.752197 27792 tablet_replica.cc:330] T a3d0d1ef1cdb409b85440c892e493e25 P af6ca90f9bf440aab1d4d494162dbd1f: stopping tablet replica
I20260928 06:34:13.752300 27792 raft_consensus.cc:2229] T a3d0d1ef1cdb409b85440c892e493e25 P af6ca90f9bf440aab1d4d494162dbd1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:34:13.752396 27792 raft_consensus.cc:2258] T a3d0d1ef1cdb409b85440c892e493e25 P af6ca90f9bf440aab1d4d494162dbd1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:34:13.752441 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 27352
I20260928 06:34:13.753116 27790 tablet_replica.cc:330] T a3d0d1ef1cdb409b85440c892e493e25 P 3e73989113cb44ee949db46590ad401c: stopping tablet replica
I20260928 06:34:13.753213 27790 raft_consensus.cc:2229] T a3d0d1ef1cdb409b85440c892e493e25 P 3e73989113cb44ee949db46590ad401c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:34:13.753309 27790 raft_consensus.cc:2258] T a3d0d1ef1cdb409b85440c892e493e25 P 3e73989113cb44ee949db46590ad401c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:34:13.753866 27792 ts_tablet_manager.cc:1905] T a3d0d1ef1cdb409b85440c892e493e25 P af6ca90f9bf440aab1d4d494162dbd1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:34:13.753875 27790 ts_tablet_manager.cc:1905] T a3d0d1ef1cdb409b85440c892e493e25 P 3e73989113cb44ee949db46590ad401c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:34:13.755827 27790 ts_tablet_manager.cc:1918] T a3d0d1ef1cdb409b85440c892e493e25 P 3e73989113cb44ee949db46590ad401c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:34:13.755821 27792 ts_tablet_manager.cc:1918] T a3d0d1ef1cdb409b85440c892e493e25 P af6ca90f9bf440aab1d4d494162dbd1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:34:13.755910 27790 log.cc:1196] T a3d0d1ef1cdb409b85440c892e493e25 P 3e73989113cb44ee949db46590ad401c: Deleting WAL directory at /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/minicluster-data/ts-2/wal/wals/a3d0d1ef1cdb409b85440c892e493e25
I20260928 06:34:13.755908 27792 log.cc:1196] T a3d0d1ef1cdb409b85440c892e493e25 P af6ca90f9bf440aab1d4d494162dbd1f: Deleting WAL directory at /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577196946045-17013-0/minicluster-data/ts-1/wal/wals/a3d0d1ef1cdb409b85440c892e493e25
I20260928 06:34:13.756215 27790 ts_tablet_manager.cc:1939] T a3d0d1ef1cdb409b85440c892e493e25 P 3e73989113cb44ee949db46590ad401c: Deleting consensus metadata
I20260928 06:34:13.756222 27792 ts_tablet_manager.cc:1939] T a3d0d1ef1cdb409b85440c892e493e25 P af6ca90f9bf440aab1d4d494162dbd1f: Deleting consensus metadata
I20260928 06:34:13.756668 27276 catalog_manager.cc:4841] TS af6ca90f9bf440aab1d4d494162dbd1f (127.16.157.66:33681): tablet a3d0d1ef1cdb409b85440c892e493e25 (table test-table [id=f80ea976cd8042c99e3285e8d7c7af4d]) successfully deleted
I20260928 06:34:13.756793 27275 catalog_manager.cc:4841] TS 3e73989113cb44ee949db46590ad401c (127.16.157.67:40217): tablet a3d0d1ef1cdb409b85440c892e493e25 (table test-table [id=f80ea976cd8042c99e3285e8d7c7af4d]) successfully deleted
W20260928 06:34:13.760210 27275 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.157.65:42721 (error 108)
I20260928 06:34:13.760422 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 27486
I20260928 06:34:13.762149 27765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:13.762013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.65:42721 (local address 127.0.0.1:53550)
0928 06:34:13.762072 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:13.762122 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.65:42721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 06:34:13.762290 27275 catalog_manager.cc:4584] TS dd4d3078692b43fe83340dfb79547f33 (127.16.157.65:42721): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a3d0d1ef1cdb409b85440c892e493e25: Network error: Client connection negotiation failed: client connection to 127.16.157.65:42721: connect: Connection refused (error 111)
I20260928 06:34:13.765096 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 27621
I20260928 06:34:13.770413 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 27257
2026-09-28T06:34:13Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2109 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:34:13 dist-test-slave-rrj5 krb5kdc[27796](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:34:13 dist-test-slave-rrj5 krb5kdc[27796](info): set up 2 sockets
Sep 28 06:34:13 dist-test-slave-rrj5 krb5kdc[27796](info): commencing operation
krb5kdc: starting...
W20260928 06:34:14.899950 17013 mini_kdc.cc:121] Time spent starting KDC: real 1.108s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Sep 28 06:34:14 dist-test-slave-rrj5 krb5kdc[27796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:34:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:34:15Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.16.157.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.16.157.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.16.157.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/krb5kdc/rangeradmin_127.16.157.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.16.157.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/krb5kdc/rangeradmin_127.16.157.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.16.157.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.16.157.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.16.157.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/krb5kdc/rangerlookup_127.16.157.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.16.157.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/krb5kdc/rangerlookup_127.16.157.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.157.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.16.157.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.157.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/krb5kdc/HTTP_127.16.157.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.157.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/krb5kdc/HTTP_127.16.157.84@KRBTEST.COM.keytab.
I20260928 06:34:15.080243 17013 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskGDaHLl/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/postgres -l logfile start
2026-09-28 06:34:17.841 UTC [27826] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 06:34:17.841 UTC [27826] LOG: listening on IPv4 address "127.16.157.84", port 36869
2026-09-28 06:34:17.845 UTC [27826] LOG: listening on Unix socket "/tmp/.s.PGSQL.36869"
2026-09-28 06:34:17.874 UTC [27829] LOG: database system was shut down at 2026-09-28 06:34:16 UTC
2026-09-28 06:34:17.878 UTC [27826] LOG: database system is ready to accept connections
I20260928 06:34:18.907642 17013 mini_postgres.cc:96] Postgres bound to 36869
2026-09-28 06:34:18.918 UTC [27837] FATAL: database "slave" does not exist
127.16.157.84:36869 - accepting connections
I20260928 06:34:18.919075 17013 mini_ranger.cc:161] Starting Ranger...
I20260928 06:34:18.940196 17013 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 06:34:19.442651 17013 mini_ranger.cc:87] Created ranger Postgres database
I20260928 06:34:19.442740 17013 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/ranger-admin
2026-09-28 06:34:20,133 [I] DB FLAVOR :POSTGRES
2026-09-28 06:34:20,133 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 06:34:20,133 [I] Checking connection..
2026-09-28 06:34:20,133 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGDaHLl/build/release/bin/postgresql.jar:/tmp/dist-test-taskGDaHLl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.157.84:36869/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 06:34:20,459 [I] Checking connection passed.
2026-09-28 06:34:20,460 [I] --------- Verifying version history table ---------
2026-09-28 06:34:20,460 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGDaHLl/build/release/bin/postgresql.jar:/tmp/dist-test-taskGDaHLl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.157.84:36869/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 06:34:20,764 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 06:34:20,765 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGDaHLl/build/release/bin/postgresql.jar:/tmp/dist-test-taskGDaHLl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.157.84:36869/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 06:34:21,057 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 06:34:21,058 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 06:34:21,058 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGDaHLl/build/release/bin/postgresql.jar:/tmp/dist-test-taskGDaHLl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.157.84:36869/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGDaHLl/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 06:34:21.438 UTC [27901] WARNING: there is no transaction in progress
2026-09-28 06:34:21,445 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 06:34:21,445 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGDaHLl/build/release/bin/postgresql.jar:/tmp/dist-test-taskGDaHLl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.157.84:36869/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 06:34:21,736 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 06:34:21,736 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 06:34:21,736 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGDaHLl/build/release/bin/postgresql.jar:/tmp/dist-test-taskGDaHLl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.157.84:36869/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 06:34:22,038 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGDaHLl/build/release/bin/postgresql.jar:/tmp/dist-test-taskGDaHLl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.157.84:36869/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 06:34:22,315 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGDaHLl/build/release/bin/postgresql.jar:/tmp/dist-test-taskGDaHLl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.157.84:36869/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-rrj5.c.gcp-upstream.internal','N') ;"
2026-09-28 06:34:22,604 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGDaHLl/build/release/bin/postgresql.jar:/tmp/dist-test-taskGDaHLl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.157.84:36869/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 06:34:22,885 [I] Table x_portal_user does not exist in database ranger
2026-09-28 06:34:22,886 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGDaHLl/build/release/bin/postgresql.jar:/tmp/dist-test-taskGDaHLl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.157.84:36869/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 06:34:23,171 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 06:34:23,171 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGDaHLl/build/release/bin/postgresql.jar:/tmp/dist-test-taskGDaHLl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.157.84:36869/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 06:34:23,477 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGDaHLl/build/release/bin/postgresql.jar:/tmp/dist-test-taskGDaHLl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.157.84:36869/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 06:34:23,753 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 06:34:23,753 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGDaHLl/build/release/bin/postgresql.jar:/tmp/dist-test-taskGDaHLl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.157.84:36869/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGDaHLl/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 06:34:24.784 UTC [28010] WARNING: there is no transaction in progress
2026-09-28 06:34:24.801 UTC [28010] WARNING: there is no transaction in progress
2026-09-28 06:34:24.820 UTC [28010] WARNING: there is no transaction in progress
2026-09-28 06:34:24.838 UTC [28010] WARNING: there is no transaction in progress
2026-09-28 06:34:24.854 UTC [28010] WARNING: there is no transaction in progress
2026-09-28 06:34:24.874 UTC [28010] WARNING: there is no transaction in progress
2026-09-28 06:34:24.932 UTC [28010] WARNING: there is no transaction in progress
2026-09-28 06:34:24.939 UTC [28010] WARNING: there is no transaction in progress
2026-09-28 06:34:24.954 UTC [28010] WARNING: there is no transaction in progress
2026-09-28 06:34:24.966 UTC [28010] WARNING: there is no transaction in progress
2026-09-28 06:34:24.978 UTC [28010] WARNING: there is no transaction in progress
2026-09-28 06:34:24.993 UTC [28010] WARNING: there is no transaction in progress
2026-09-28 06:34:25.002 UTC [28010] WARNING: there is no transaction in progress
2026-09-28 06:34:25.008 UTC [28010] WARNING: there is no transaction in progress
2026-09-28 06:34:25.016 UTC [28010] WARNING: there is no transaction in progress
2026-09-28 06:34:25.023 UTC [28010] WARNING: there is no transaction in progress
2026-09-28 06:34:25,669 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 06:34:25,670 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGDaHLl/build/release/bin/postgresql.jar:/tmp/dist-test-taskGDaHLl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.157.84:36869/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 06:34:25,932 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 06:34:25,932 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGDaHLl/build/release/bin/postgresql.jar:/tmp/dist-test-taskGDaHLl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.157.84:36869/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 06:34:26,215 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 06:34:26,216 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGDaHLl/build/release/bin/postgresql.jar:/tmp/dist-test-taskGDaHLl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.157.84:36869/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 06:34:26,486 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 06:34:26,487 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGDaHLl/build/release/bin/postgresql.jar:/tmp/dist-test-taskGDaHLl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.157.84:36869/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 06:34:26,782 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGDaHLl/build/release/bin/postgresql.jar:/tmp/dist-test-taskGDaHLl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.157.84:36869/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 06:34:27,077 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGDaHLl/build/release/bin/postgresql.jar:/tmp/dist-test-taskGDaHLl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.157.84:36869/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-rrj5.c.gcp-upstream.internal';"
2026-09-28 06:34:27,348 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 06:34:27,348 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGDaHLl/build/release/bin/postgresql.jar:/tmp/dist-test-taskGDaHLl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.157.84:36869/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 06:34:27,630 [I] DB_PATCHES have already been applied
I20260928 06:34:27.637753 17013 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/ranger-admin:/tmp/dist-test-taskGDaHLl/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskGDaHLl/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskGDaHLl/build/release/bin/postgresql.jar:/tmp/dist-test-taskGDaHLl/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 06:34:27.637832 17013 mini_ranger.cc:193] Using host: 127.16.157.84
I20260928 06:34:27.641081 17013 mini_ranger.cc:227] Ranger admin URL: http://127.16.157.84:39083
Sep 28, 2026 6:34:28 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 6:34:28 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 6:34:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 6:34:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 6:34:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 6:34:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 6:34:28 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 06:34:29.934754 17013 mini_ranger.cc:160] Time spent starting Ranger: real 11.016s user 0.000s sys 0.011s
Sep 28, 2026 6:34:52 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 6:34:52 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 6:34:52 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 6:34:52 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 6:34:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 6:34:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 6:34:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 6:34:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 6:34:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 6:34:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 6:34:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 6:34:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 6:34:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 6:34:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 6:34:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 6:34:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 6:34:52 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 6:34:53 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 06:34:54.618251 17013 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.16.157.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.16.157.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.16.157.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/krb5kdc/rangerkms_127.16.157.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.16.157.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/krb5kdc/rangerkms_127.16.157.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.157.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.16.157.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.157.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/krb5kdc/HTTP_127.16.157.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.157.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/krb5kdc/HTTP_127.16.157.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 06:34:54 dist-test-slave-rrj5 krb5kdc[27796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577294, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 06:34:54.700951 17013 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 06:34:54.711439 17013 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 06:34:55.390666 17013 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 06:34:55.390765 17013 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/ranger-kms
I20260928 06:34:55.390789 17013 mini_ranger_kms.cc:227] Using postgres at 127.16.157.84:36869
2026-09-28 06:34:55,983 [I] DB FLAVOR :POSTGRES
2026-09-28 06:34:55,983 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 06:34:55,983 [I] Checking connection
2026-09-28 06:34:55,983 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGDaHLl/build/release/bin/postgresql.jar:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.157.84:36869/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 06:34:56,212 [I] connection success
2026-09-28 06:34:56,212 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 06:34:56,212 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 06:34:56,212 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGDaHLl/build/release/bin/postgresql.jar:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.157.84:36869/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 06:34:56,453 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 06:34:56,453 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 06:34:56,453 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 06:34:56,453 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGDaHLl/build/release/bin/postgresql.jar:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.157.84:36869/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 06:34:56,694 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 06:34:57.165294 17013 mini_ranger_kms.cc:310] Created kms service
I20260928 06:34:57.329752 17013 mini_ranger_kms.cc:326] Created kudu user
I20260928 06:34:57.381054 17013 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 06:34:57 dist-test-slave-rrj5 krb5kdc[27796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577294, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.16.157.84@KRBTEST.COM
I20260928 06:34:57.790196 17013 mini_ranger_kms.cc:384] Added ranger policy
I20260928 06:34:57.790318 17013 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/ranger-kms:/tmp/dist-test-taskGDaHLl/build/release/bin/postgresql.jar:/tmp/dist-test-taskGDaHLl/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskGDaHLl/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 06:34:57.790338 17013 mini_ranger_kms.cc:240] Using host: 127.16.157.84
I20260928 06:34:57.794775 17013 mini_ranger_kms.cc:276] Ranger KMS PID: 28193
I20260928 06:34:57.794899 17013 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.16.157.84:44333
Sep 28, 2026 6:34:58 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 6:34:58 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 6:34:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 6:34:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 6:34:59 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 6:34:59 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 28 06:34:59 dist-test-slave-rrj5 krb5kdc[27796](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790577299, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.157.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:00.023090 17013 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.322s user 0.023s sys 0.205s
I20260928 06:35:00.023207 17013 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 06:35:00.023291 17013 mini_ranger_kms.cc:401] 127.16.157.84:44333/kms/v1/keys
Sep 28 06:35:00 dist-test-slave-rrj5 krb5kdc[27796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577299, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.157.84@KRBTEST.COM for HTTP/127.16.157.84@KRBTEST.COM
Sep 28 06:35:00 dist-test-slave-rrj5 krb5kdc[27796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577299, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.157.84@KRBTEST.COM for HTTP/127.16.157.84@KRBTEST.COM
Sep 28 06:35:02 dist-test-slave-rrj5 krb5kdc[27796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.16.157.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab.
Entry for principal kudu/127.16.157.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab.
I20260928 06:35:02.889937 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:42009
--webserver_interface=127.16.157.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=kudu/127.16.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:36315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.157.126:42009
--ranger_config_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.157.84:44333/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:02.986394 28243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:02.986585 28243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:02.986605 28243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:02.987732 28243 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:02.987773 28243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:02.987794 28243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:02.987807 28243 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:02.987819 28243 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:35:02.989285 28243 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:36315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/master-0/wal
--ranger_kms_url=127.16.157.84:44333/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.16.157.126:42009
--ranger_config_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=kudu/127.16.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:42009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28243
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:35:02.989512 28243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:02.989789 28243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:02.994475 28248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:02.994462 28251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:02.994462 28249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:02.994973 28243 server_base.cc:1034] running on GCE node
I20260928 06:35:02.995221 28243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:02.995496 28243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:02.996655 28243 hybrid_clock.cc:648] HybridClock initialized: now 1790577302996643 us; error 30 us; skew 500 ppm
Sep 28 06:35:02 dist-test-slave-rrj5 krb5kdc[27796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577302, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:02.999228 28243 init.cc:377] Logged in from keytab as kudu/127.16.157.126@KRBTEST.COM (short username kudu)
I20260928 06:35:02.999737 28243 webserver.cc:460] Webserver started at http://127.16.157.126:43645/ using document root <none> and password file <none>
I20260928 06:35:02.999948 28243 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:03.000007 28243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:03.000142 28243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 06:35:02 dist-test-slave-rrj5 krb5kdc[27796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577302, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.126@KRBTEST.COM for HTTP/127.16.157.84@KRBTEST.COM
I20260928 06:35:03.048010 28243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/master-0/data/instance:
uuid: "7e7a05cb5d36487bb01a925538c66b16"
format_stamp: "Formatted at 2026-09-28 06:35:03 on dist-test-slave-rrj5"
server_key: "2837f518281a4c6bb3e5cf2970057f35"
server_key_iv: "edb3f093521d18421a596d1c4a691003"
server_key_version: "kuduclusterkey@0"
I20260928 06:35:03.048481 28243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/master-0/wal/instance:
uuid: "7e7a05cb5d36487bb01a925538c66b16"
format_stamp: "Formatted at 2026-09-28 06:35:03 on dist-test-slave-rrj5"
server_key: "2837f518281a4c6bb3e5cf2970057f35"
server_key_iv: "edb3f093521d18421a596d1c4a691003"
server_key_version: "kuduclusterkey@0"
I20260928 06:35:03.050410 28243 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260928 06:35:03.064561 28258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:03.064817 28243 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:35:03.064934 28243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/master-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/master-0/wal
uuid: "7e7a05cb5d36487bb01a925538c66b16"
format_stamp: "Formatted at 2026-09-28 06:35:03 on dist-test-slave-rrj5"
server_key: "2837f518281a4c6bb3e5cf2970057f35"
server_key_iv: "edb3f093521d18421a596d1c4a691003"
server_key_version: "kuduclusterkey@0"
I20260928 06:35:03.065037 28243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:03.080986 28243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:03.083271 28243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:03.083437 28243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:03.088562 28243 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.126:42009
I20260928 06:35:03.088632 28310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.126:42009 every 8 connection(s)
I20260928 06:35:03.088999 28243 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
I20260928 06:35:03.089794 28311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:03.092114 28311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e7a05cb5d36487bb01a925538c66b16: Bootstrap starting.
I20260928 06:35:03.092705 28311 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e7a05cb5d36487bb01a925538c66b16: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:03.092999 28311 log.cc:824] T 00000000000000000000000000000000 P 7e7a05cb5d36487bb01a925538c66b16: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:03.094693 28311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e7a05cb5d36487bb01a925538c66b16: No bootstrap required, opened a new log
I20260928 06:35:03.096546 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 28243
I20260928 06:35:03.096673 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/master-0/wal/instance
I20260928 06:35:03.096918 17013 external_mini_cluster.cc:1428] Setting key 021ddf320230664199cfe5035a2f551f
I20260928 06:35:03.097221 28311 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e7a05cb5d36487bb01a925538c66b16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e7a05cb5d36487bb01a925538c66b16" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 42009 } }
I20260928 06:35:03.097440 28311 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e7a05cb5d36487bb01a925538c66b16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:03.097486 28311 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e7a05cb5d36487bb01a925538c66b16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e7a05cb5d36487bb01a925538c66b16, State: Initialized, Role: FOLLOWER
I20260928 06:35:03.097646 28311 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e7a05cb5d36487bb01a925538c66b16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e7a05cb5d36487bb01a925538c66b16" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 42009 } }
I20260928 06:35:03.097748 28311 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e7a05cb5d36487bb01a925538c66b16 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:35:03.097791 28311 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e7a05cb5d36487bb01a925538c66b16 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:35:03.097851 28311 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e7a05cb5d36487bb01a925538c66b16 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:03.098623 28311 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e7a05cb5d36487bb01a925538c66b16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e7a05cb5d36487bb01a925538c66b16" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 42009 } }
I20260928 06:35:03.098753 28311 leader_election.cc:302] T 00000000000000000000000000000000 P 7e7a05cb5d36487bb01a925538c66b16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e7a05cb5d36487bb01a925538c66b16; no voters:
I20260928 06:35:03.098951 28311 leader_election.cc:288] T 00000000000000000000000000000000 P 7e7a05cb5d36487bb01a925538c66b16 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:35:03.098978 28316 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e7a05cb5d36487bb01a925538c66b16 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:03.099237 28316 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e7a05cb5d36487bb01a925538c66b16 [term 1 LEADER]: Becoming Leader. State: Replica: 7e7a05cb5d36487bb01a925538c66b16, State: Running, Role: LEADER
I20260928 06:35:03.099239 28311 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e7a05cb5d36487bb01a925538c66b16 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:35:03.099471 28316 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e7a05cb5d36487bb01a925538c66b16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e7a05cb5d36487bb01a925538c66b16" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 42009 } }
I20260928 06:35:03.099876 28316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e7a05cb5d36487bb01a925538c66b16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e7a05cb5d36487bb01a925538c66b16. Latest consensus state: current_term: 1 leader_uuid: "7e7a05cb5d36487bb01a925538c66b16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e7a05cb5d36487bb01a925538c66b16" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 42009 } } }
I20260928 06:35:03.100044 28316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e7a05cb5d36487bb01a925538c66b16 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:03.100313 28325 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:35:03.100932 28325 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:35:03.101181 28311 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 06:35:03.101619 28318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e7a05cb5d36487bb01a925538c66b16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e7a05cb5d36487bb01a925538c66b16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e7a05cb5d36487bb01a925538c66b16" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 42009 } } }
I20260928 06:35:03.101708 28318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e7a05cb5d36487bb01a925538c66b16 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:03.103848 28325 catalog_manager.cc:1347] Generated new cluster ID: 723c6f1987b3493c9e36f330b37f039e
I20260928 06:35:03.103902 28325 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:35:03 dist-test-slave-rrj5 krb5kdc[27796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:35:03.119725 28315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:03.097456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40542 (local address 127.16.157.126:42009)
0928 06:35:03.097697 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:03.097706 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:03.097758 (+ 52us) server_negotiation.cc:407] Connection header received
0928 06:35:03.098056 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:03.098063 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:03.098131 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:03.098315 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:03.099090 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:03.100548 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:03.101665 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:03.102255 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:03.102331 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:03.105743 (+ 3412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:03.105747 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:03.105747 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:03.105759 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:03.112965 (+ 7206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:03.118000 (+ 5035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:03.118004 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:03.118004 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:03.118126 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:03.118386 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:03.118387 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:03.118387 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:03.118829 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:03.119025 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:03.119311 (+ 286us) server_negotiation.cc:299] Negotiation successful
0928 06:35:03.119493 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":85,"threads_started":1}
I20260928 06:35:03.135597 28325 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:35:03.136309 28325 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:35:03.140925 28325 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e7a05cb5d36487bb01a925538c66b16: Generated new TSK 0
I20260928 06:35:03.141079 28325 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.157.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab.
Entry for principal kudu/127.16.157.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab.
I20260928 06:35:03.836382 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.157.84:44333/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.65:0
--local_ip_for_outbound_sockets=127.16.157.65
--webserver_interface=127.16.157.65
--webserver_port=0
--tserver_master_addrs=127.16.157.126:42009
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab
--principal=kudu/127.16.157.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:36315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:03.943676 28356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:03.943946 28356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:03.943979 28356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:03.945663 28356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:03.945757 28356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:03.945825 28356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.65
I20260928 06:35:03.947964 28356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:36315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.16.157.84:44333/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab
--principal=kudu/127.16.157.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.157.65
--webserver_port=0
--tserver_master_addrs=127.16.157.126:42009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28356
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.65
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:35:03.948334 28356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:03.948648 28356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:03.954296 28362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:03.954336 28361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:03.954618 28364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:03.954784 28356 server_base.cc:1034] running on GCE node
I20260928 06:35:03.954978 28356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:03.955245 28356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:03.956410 28356 hybrid_clock.cc:648] HybridClock initialized: now 1790577303956379 us; error 46 us; skew 500 ppm
Sep 28 06:35:03 dist-test-slave-rrj5 krb5kdc[27796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577303, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:03.964781 28356 init.cc:377] Logged in from keytab as kudu/127.16.157.65@KRBTEST.COM (short username kudu)
I20260928 06:35:03.965400 28356 webserver.cc:460] Webserver started at http://127.16.157.65:46031/ using document root <none> and password file <none>
I20260928 06:35:03.965757 28356 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:03.965839 28356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:03.965987 28356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 06:35:03 dist-test-slave-rrj5 krb5kdc[27796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577303, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.65@KRBTEST.COM for HTTP/127.16.157.84@KRBTEST.COM
I20260928 06:35:03.980716 28356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/ts-0/data/instance:
uuid: "b036ed2625554b3ba52c0a9e27a5fc4e"
format_stamp: "Formatted at 2026-09-28 06:35:03 on dist-test-slave-rrj5"
server_key: "47029738349660dad6f92a4f639d426d"
server_key_iv: "4374d8e5b5649bf6bb447f56d39b2b81"
server_key_version: "kuduclusterkey@0"
I20260928 06:35:03.981118 28356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/ts-0/wal/instance:
uuid: "b036ed2625554b3ba52c0a9e27a5fc4e"
format_stamp: "Formatted at 2026-09-28 06:35:03 on dist-test-slave-rrj5"
server_key: "47029738349660dad6f92a4f639d426d"
server_key_iv: "4374d8e5b5649bf6bb447f56d39b2b81"
server_key_version: "kuduclusterkey@0"
I20260928 06:35:03.985989 28356 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.003s
I20260928 06:35:04.002908 28373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:04.003129 28356 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:35:04.003238 28356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/ts-0/wal
uuid: "b036ed2625554b3ba52c0a9e27a5fc4e"
format_stamp: "Formatted at 2026-09-28 06:35:03 on dist-test-slave-rrj5"
server_key: "47029738349660dad6f92a4f639d426d"
server_key_iv: "4374d8e5b5649bf6bb447f56d39b2b81"
server_key_version: "kuduclusterkey@0"
I20260928 06:35:04.003458 28356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:04.026830 28356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:04.029279 28356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:04.029496 28356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:04.029850 28356 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:04.030262 28356 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:04.030355 28356 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:04.030459 28356 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:04.030544 28356 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:04.037562 28356 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.65:33807
I20260928 06:35:04.038230 28486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.65:33807 every 8 connection(s)
I20260928 06:35:04.039546 28356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
I20260928 06:35:04.045262 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 28356
I20260928 06:35:04.045372 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/ts-0/wal/instance
I20260928 06:35:04.045605 17013 external_mini_cluster.cc:1428] Setting key 6d28bd121ebc4af0fcd3006549b76847
Sep 28 06:35:04 dist-test-slave-rrj5 krb5kdc[27796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577303, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.65@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:35:04.056325 28490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:04.040624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:59049 (local address 127.16.157.126:42009)
0928 06:35:04.045395 (+ 4771us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:04.045398 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:04.045408 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:35:04.045489 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:04.045490 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:04.045505 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:04.045613 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:04.047024 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:04.047780 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:04.049298 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:04.049642 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:04.049712 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:04.052494 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:04.052496 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:04.052496 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:04.052504 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:04.054490 (+ 1986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:04.055055 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:04.055056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:04.055057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:04.055133 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:04.055446 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:04.055447 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:04.055448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:04.055606 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:04.055672 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:04.056110 (+ 438us) server_negotiation.cc:299] Negotiation successful
0928 06:35:04.056206 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4669,"thread_start_us":80,"threads_started":1}
I20260928 06:35:04.058275 28489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:04.040325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:42009 (local address 127.16.157.65:59049)
0928 06:35:04.044432 (+ 4107us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:04.044481 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:04.045119 (+ 638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:04.045802 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:04.045807 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:04.046573 (+ 766us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:04.046880 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:04.046888 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:04.047908 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:04.047909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:04.049187 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:04.049190 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:04.049798 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:04.049799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:04.049964 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:04.050566 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:04.050576 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:04.052369 (+ 1793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:04.054601 (+ 2232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:04.054609 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:04.054611 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:04.054948 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:04.055217 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:04.055218 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:04.055219 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:04.055358 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:04.055714 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:04.055721 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:04.055933 (+ 212us) client_negotiation.cc:769] Sending connection context
0928 06:35:04.056084 (+ 151us) client_negotiation.cc:240] Negotiation successful
0928 06:35:04.056242 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4057,"thread_start_us":89,"threads_started":1}
I20260928 06:35:04.059103 28487 heartbeater.cc:347] Connected to a master server at 127.16.157.126:42009
I20260928 06:35:04.059235 28487 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:04.059442 28487 heartbeater.cc:511] Master 127.16.157.126:42009 requested a full tablet report, sending...
I20260928 06:35:04.060078 28275 ts_manager.cc:194] Registered new tserver with Master: b036ed2625554b3ba52c0a9e27a5fc4e (127.16.157.65:33807)
I20260928 06:35:04.060984 28275 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.157.65@KRBTEST.COM'} at 127.16.157.65:59049
WARNING: no policy specified for kudu/127.16.157.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab.
Entry for principal kudu/127.16.157.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab.
I20260928 06:35:04.084098 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.157.84:44333/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.66:0
--local_ip_for_outbound_sockets=127.16.157.66
--webserver_interface=127.16.157.66
--webserver_port=0
--tserver_master_addrs=127.16.157.126:42009
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab
--principal=kudu/127.16.157.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:36315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 06:35:04 dist-test-slave-rrj5 krb5kdc[27796](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790577304, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 06:35:04.201135 28495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:04.201395 28495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:04.201427 28495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:04.202703 28495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:04.202781 28495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:04.202831 28495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.66
I20260928 06:35:04.204355 28495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:36315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.16.157.84:44333/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab
--principal=kudu/127.16.157.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.157.66
--webserver_port=0
--tserver_master_addrs=127.16.157.126:42009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28495
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.66
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:35:04.204619 28495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:04.204845 28495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:04.210752 28501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:04.210847 28503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:04.210744 28500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:04.211436 28495 server_base.cc:1034] running on GCE node
I20260928 06:35:04.211745 28495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:04.212031 28495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:04.213218 28495 hybrid_clock.cc:648] HybridClock initialized: now 1790577304213192 us; error 42 us; skew 500 ppm
Sep 28 06:35:04 dist-test-slave-rrj5 krb5kdc[27796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577304, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:04.216331 28495 init.cc:377] Logged in from keytab as kudu/127.16.157.66@KRBTEST.COM (short username kudu)
I20260928 06:35:04.216811 28495 webserver.cc:460] Webserver started at http://127.16.157.66:41391/ using document root <none> and password file <none>
I20260928 06:35:04.217056 28495 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:04.217128 28495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:04.217286 28495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 06:35:04 dist-test-slave-rrj5 krb5kdc[27796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577304, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.66@KRBTEST.COM for HTTP/127.16.157.84@KRBTEST.COM
I20260928 06:35:04.239009 28495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/ts-1/data/instance:
uuid: "0d84a82f440d4532b362d4b985579b03"
format_stamp: "Formatted at 2026-09-28 06:35:04 on dist-test-slave-rrj5"
server_key: "f4324b66d95bcd99a12f53204cde63a0"
server_key_iv: "0caf1075cae461163150338a9386e1fb"
server_key_version: "kuduclusterkey@0"
I20260928 06:35:04.239632 28495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/ts-1/wal/instance:
uuid: "0d84a82f440d4532b362d4b985579b03"
format_stamp: "Formatted at 2026-09-28 06:35:04 on dist-test-slave-rrj5"
server_key: "f4324b66d95bcd99a12f53204cde63a0"
server_key_iv: "0caf1075cae461163150338a9386e1fb"
server_key_version: "kuduclusterkey@0"
I20260928 06:35:04.242008 28495 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260928 06:35:04.257889 28511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:04.258103 28495 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:35:04.258188 28495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/ts-1/wal
uuid: "0d84a82f440d4532b362d4b985579b03"
format_stamp: "Formatted at 2026-09-28 06:35:04 on dist-test-slave-rrj5"
server_key: "f4324b66d95bcd99a12f53204cde63a0"
server_key_iv: "0caf1075cae461163150338a9386e1fb"
server_key_version: "kuduclusterkey@0"
I20260928 06:35:04.258266 28495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:04.274964 28495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:04.277645 28495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:04.277806 28495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:04.278041 28495 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:04.278421 28495 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:04.278486 28495 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:04.278518 28495 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:04.278544 28495 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:04.286288 28495 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.66:40087
I20260928 06:35:04.286377 28624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.66:40087 every 8 connection(s)
I20260928 06:35:04.286717 28495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
I20260928 06:35:04.293807 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 28495
I20260928 06:35:04.293936 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/ts-1/wal/instance
Sep 28 06:35:04 dist-test-slave-rrj5 krb5kdc[27796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577304, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.66@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:35:04.294121 17013 external_mini_cluster.cc:1428] Setting key de18614cf371e7b38b05790a66f4498a
I20260928 06:35:04.297649 28490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:04.287523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.66:39157 (local address 127.16.157.126:42009)
0928 06:35:04.287569 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:04.287571 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:04.287808 (+ 237us) server_negotiation.cc:407] Connection header received
0928 06:35:04.288350 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:04.288352 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:04.288370 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:04.288427 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:04.289036 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:04.289562 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:04.290907 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:04.291198 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:04.291245 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:04.294687 (+ 3442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:04.294688 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:04.294688 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:04.294695 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:04.296174 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:04.296584 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:04.296585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:04.296585 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:04.296648 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:04.296899 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:04.296900 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:04.296918 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:04.297040 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:04.297114 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:04.297434 (+ 320us) server_negotiation.cc:299] Negotiation successful
0928 06:35:04.297521 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 06:35:04.297900 28627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:04.287506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:42009 (local address 127.16.157.66:39157)
0928 06:35:04.287675 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:04.287710 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:04.288223 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:04.288503 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:04.288506 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:04.288692 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:04.288940 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:04.288945 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:04.289694 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:04.289695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:04.290797 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:04.290800 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:04.292005 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:04.292006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:04.292148 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:04.292763 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:04.292773 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:04.294590 (+ 1817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:04.296283 (+ 1693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:04.296290 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:04.296292 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:04.296506 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:04.296715 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:04.296716 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:04.296716 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:04.296830 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:04.297114 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:04.297127 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:04.297298 (+ 171us) client_negotiation.cc:769] Sending connection context
0928 06:35:04.297413 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 06:35:04.297555 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":85,"threads_started":1}
I20260928 06:35:04.298326 28625 heartbeater.cc:347] Connected to a master server at 127.16.157.126:42009
I20260928 06:35:04.298449 28625 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:04.298630 28625 heartbeater.cc:511] Master 127.16.157.126:42009 requested a full tablet report, sending...
I20260928 06:35:04.299031 28275 ts_manager.cc:194] Registered new tserver with Master: 0d84a82f440d4532b362d4b985579b03 (127.16.157.66:40087)
I20260928 06:35:04.299477 28275 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.157.66@KRBTEST.COM'} at 127.16.157.66:39157
WARNING: no policy specified for kudu/127.16.157.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab.
Entry for principal kudu/127.16.157.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab.
I20260928 06:35:04.323971 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.157.84:44333/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.67:0
--local_ip_for_outbound_sockets=127.16.157.67
--webserver_interface=127.16.157.67
--webserver_port=0
--tserver_master_addrs=127.16.157.126:42009
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab
--principal=kudu/127.16.157.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:36315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:04.435402 28630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:04.435771 28630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:04.435876 28630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:04.437753 28630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:04.437875 28630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:04.437964 28630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.67
I20260928 06:35:04.440322 28630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:36315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.16.157.84:44333/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab
--principal=kudu/127.16.157.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.157.67
--webserver_port=0
--tserver_master_addrs=127.16.157.126:42009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.67
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:35:04.440586 28630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:04.440915 28630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:04.447273 28636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:04.447485 28635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:04.447690 28638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:04.448071 28630 server_base.cc:1034] running on GCE node
I20260928 06:35:04.448276 28630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:04.448553 28630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:04.449716 28630 hybrid_clock.cc:648] HybridClock initialized: now 1790577304449693 us; error 37 us; skew 500 ppm
Sep 28 06:35:04 dist-test-slave-rrj5 krb5kdc[27796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577304, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:04.452913 28630 init.cc:377] Logged in from keytab as kudu/127.16.157.67@KRBTEST.COM (short username kudu)
I20260928 06:35:04.453413 28630 webserver.cc:460] Webserver started at http://127.16.157.67:43707/ using document root <none> and password file <none>
I20260928 06:35:04.453683 28630 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:04.453735 28630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:04.453856 28630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 06:35:04 dist-test-slave-rrj5 krb5kdc[27796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577304, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.67@KRBTEST.COM for HTTP/127.16.157.84@KRBTEST.COM
I20260928 06:35:04.480141 28630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/ts-2/data/instance:
uuid: "e2ad591cdb7545e3abaae800c3e9aeda"
format_stamp: "Formatted at 2026-09-28 06:35:04 on dist-test-slave-rrj5"
server_key: "d98cd0ddba448489d3c43253408e79d2"
server_key_iv: "08060a5dacc258c01a83fdb72a1cbe09"
server_key_version: "kuduclusterkey@0"
I20260928 06:35:04.480611 28630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/ts-2/wal/instance:
uuid: "e2ad591cdb7545e3abaae800c3e9aeda"
format_stamp: "Formatted at 2026-09-28 06:35:04 on dist-test-slave-rrj5"
server_key: "d98cd0ddba448489d3c43253408e79d2"
server_key_iv: "08060a5dacc258c01a83fdb72a1cbe09"
server_key_version: "kuduclusterkey@0"
I20260928 06:35:04.483525 28630 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.000s
I20260928 06:35:04.503834 28645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:04.507613 28630 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.000s sys 0.000s
I20260928 06:35:04.507736 28630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/ts-2/wal
uuid: "e2ad591cdb7545e3abaae800c3e9aeda"
format_stamp: "Formatted at 2026-09-28 06:35:04 on dist-test-slave-rrj5"
server_key: "d98cd0ddba448489d3c43253408e79d2"
server_key_iv: "08060a5dacc258c01a83fdb72a1cbe09"
server_key_version: "kuduclusterkey@0"
I20260928 06:35:04.507834 28630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:04.537156 28630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:04.540055 28630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:04.540205 28630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:04.540405 28630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:04.540753 28630 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:04.540795 28630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:04.540825 28630 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:04.540853 28630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:04.547714 28630 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.67:39621
I20260928 06:35:04.548156 28630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
I20260928 06:35:04.549677 28758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.67:39621 every 8 connection(s)
I20260928 06:35:04.556965 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 28630
Sep 28 06:35:04 dist-test-slave-rrj5 krb5kdc[27796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577304, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.67@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:35:04.557080 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/ts-2/wal/instance
I20260928 06:35:04.557252 17013 external_mini_cluster.cc:1428] Setting key f3a6faf7906eaea3f9ee18796aa453f8
I20260928 06:35:04.560858 28490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:04.550429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:42723 (local address 127.16.157.126:42009)
0928 06:35:04.550505 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:04.550508 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:04.550718 (+ 210us) server_negotiation.cc:407] Connection header received
0928 06:35:04.551353 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:04.551355 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:04.551367 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:04.551447 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:04.552188 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:04.552964 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:04.554533 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:04.554849 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:04.554918 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:04.557548 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:04.557550 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:04.557550 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:04.557558 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:04.559335 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:04.559821 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:04.559822 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:04.559822 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:04.559881 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:04.560158 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:04.560159 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:04.560159 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:04.560288 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:04.560329 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:04.560680 (+ 351us) server_negotiation.cc:299] Negotiation successful
0928 06:35:04.560764 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 06:35:04.561637 28761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:04.550379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:42009 (local address 127.16.157.67:42723)
0928 06:35:04.550571 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:04.550613 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:04.551236 (+ 623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:04.551521 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:04.551526 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:04.551780 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:04.552084 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:04.552090 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:04.553073 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:04.553074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:04.554436 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:04.554439 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:04.555000 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:04.555001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:04.555144 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:04.555770 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:04.555781 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:04.557425 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:04.559422 (+ 1997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:04.559428 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:04.559430 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:04.559723 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:04.559949 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:04.559950 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:04.559950 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:04.560085 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:04.560372 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:04.560377 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:04.560533 (+ 156us) client_negotiation.cc:769] Sending connection context
0928 06:35:04.561214 (+ 681us) client_negotiation.cc:240] Negotiation successful
0928 06:35:04.561376 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":105,"threads_started":1}
I20260928 06:35:04.562161 28759 heartbeater.cc:347] Connected to a master server at 127.16.157.126:42009
I20260928 06:35:04.562268 28759 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:04.562479 28759 heartbeater.cc:511] Master 127.16.157.126:42009 requested a full tablet report, sending...
I20260928 06:35:04.562920 28275 ts_manager.cc:194] Registered new tserver with Master: e2ad591cdb7545e3abaae800c3e9aeda (127.16.157.67:39621)
I20260928 06:35:04.563422 28275 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.157.67@KRBTEST.COM'} at 127.16.157.67:42723
I20260928 06:35:04.569046 17013 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:35:04.578687 28490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:04.570729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40546 (local address 127.16.157.126:42009)
0928 06:35:04.570769 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:04.570771 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:04.570914 (+ 143us) server_negotiation.cc:407] Connection header received
0928 06:35:04.571021 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:04.571022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:04.571032 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:04.571060 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:04.571847 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:04.572647 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:04.573779 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:04.574101 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:04.574140 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:04.575720 (+ 1580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:04.575722 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:04.575722 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:04.575729 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:04.577409 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:04.577837 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:04.577837 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:04.577838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:04.577895 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:04.578107 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:04.578108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:04.578108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:04.578239 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:04.578310 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:04.578510 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 06:35:04.578589 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:35:04.581074 28275 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40546:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:35:04.583554 28275 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:35:04.595109 28775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:04.586105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.66:40087 (local address 127.0.0.1:46170)
0928 06:35:04.587432 (+ 1327us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:04.587477 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:04.587542 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:04.590675 (+ 3133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:04.590676 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:04.590679 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:04.590726 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:04.590727 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:04.592327 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:04.592328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:04.593511 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:04.593514 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:04.594536 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:04.594537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:04.594882 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:04.594911 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 06:35:04.594957 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 06:35:04.595023 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1289,"thread_start_us":88,"threads_started":1}
I20260928 06:35:04.595902 28772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:04.586207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46170 (local address 127.16.157.66:40087)
0928 06:35:04.590507 (+ 4300us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:04.590512 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:04.590523 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:35:04.590560 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:04.590562 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:04.590577 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:04.590790 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:04.590860 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:04.592242 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:04.593664 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:04.594434 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:04.595671 (+ 1237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:04.595760 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:04.595787 (+ 27us) server_negotiation.cc:299] Negotiation successful
0928 06:35:04.595827 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4259,"thread_start_us":71,"threads_started":1}
I20260928 06:35:04.595978 28774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:04.586462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40420 (local address 127.16.157.67:39621)
0928 06:35:04.587942 (+ 1480us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:04.587947 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:04.587958 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:35:04.587997 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:04.587999 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:04.588015 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:04.588091 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:04.588348 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:04.590348 (+ 2000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:04.594287 (+ 3939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:04.595057 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:04.595131 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:04.595211 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:04.595642 (+ 431us) server_negotiation.cc:299] Negotiation successful
0928 06:35:04.595690 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1438,"thread_start_us":85,"threads_started":1}
I20260928 06:35:04.596128 28773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:04.586361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.67:39621 (local address 127.0.0.1:40420)
0928 06:35:04.587658 (+ 1297us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:04.587667 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:04.587705 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:04.588193 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:04.588197 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:04.588199 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:04.588263 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:04.588265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:04.590447 (+ 2182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:04.590448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:04.594167 (+ 3719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:04.594170 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:04.595249 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:04.595250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:04.595312 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:04.595315 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:35:04.595365 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 06:35:04.595413 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1283,"mutex_wait_us":6,"thread_start_us":78,"threads_started":1}
I20260928 06:35:04.596781 28559 tablet_service.cc:1467] Processing CreateTablet for tablet 0d4e1d8d235d4dc9a17a1c9e1b99c89f (DEFAULT_TABLE table=test-table [id=76f0861164244a9895a136fe8670c3ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:04.596975 28693 tablet_service.cc:1467] Processing CreateTablet for tablet 0d4e1d8d235d4dc9a17a1c9e1b99c89f (DEFAULT_TABLE table=test-table [id=76f0861164244a9895a136fe8670c3ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:04.597142 28559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d4e1d8d235d4dc9a17a1c9e1b99c89f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:04.597307 28693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d4e1d8d235d4dc9a17a1c9e1b99c89f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:04.601539 28778 tablet_bootstrap.cc:492] T 0d4e1d8d235d4dc9a17a1c9e1b99c89f P e2ad591cdb7545e3abaae800c3e9aeda: Bootstrap starting.
I20260928 06:35:04.602946 28778 tablet_bootstrap.cc:654] T 0d4e1d8d235d4dc9a17a1c9e1b99c89f P e2ad591cdb7545e3abaae800c3e9aeda: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:04.603150 28777 tablet_bootstrap.cc:492] T 0d4e1d8d235d4dc9a17a1c9e1b99c89f P 0d84a82f440d4532b362d4b985579b03: Bootstrap starting.
I20260928 06:35:04.604050 28778 log.cc:824] T 0d4e1d8d235d4dc9a17a1c9e1b99c89f P e2ad591cdb7545e3abaae800c3e9aeda: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:04.604930 28777 tablet_bootstrap.cc:654] T 0d4e1d8d235d4dc9a17a1c9e1b99c89f P 0d84a82f440d4532b362d4b985579b03: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:04.605137 28771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:04.586551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.65:33807 (local address 127.0.0.1:35798)
0928 06:35:04.587780 (+ 1229us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:04.587788 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:04.587821 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:04.601178 (+ 13357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:04.601179 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:04.601181 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:04.601221 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:04.601222 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:04.602524 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:04.602525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:04.603678 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:04.603680 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:04.604625 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:04.604627 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:04.604690 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:04.604981 (+ 291us) client_negotiation.cc:769] Sending connection context
0928 06:35:04.605010 (+ 29us) client_negotiation.cc:240] Negotiation successful
0928 06:35:04.605054 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1217,"thread_start_us":78,"threads_started":1}
I20260928 06:35:04.605518 28776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:04.588396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35798 (local address 127.16.157.65:33807)
0928 06:35:04.600938 (+ 12542us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:04.600943 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:04.600955 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:35:04.601002 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:04.601004 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:04.601022 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:04.601100 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:04.601293 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:04.602421 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:04.603783 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:04.604527 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:04.605307 (+ 780us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:04.605377 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:04.605403 (+ 26us) server_negotiation.cc:299] Negotiation successful
0928 06:35:04.605443 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12506,"thread_start_us":72,"threads_started":1}
I20260928 06:35:04.605878 28777 log.cc:824] T 0d4e1d8d235d4dc9a17a1c9e1b99c89f P 0d84a82f440d4532b362d4b985579b03: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:04.606302 28421 tablet_service.cc:1467] Processing CreateTablet for tablet 0d4e1d8d235d4dc9a17a1c9e1b99c89f (DEFAULT_TABLE table=test-table [id=76f0861164244a9895a136fe8670c3ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:04.606609 28421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d4e1d8d235d4dc9a17a1c9e1b99c89f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:04.606678 28778 tablet_bootstrap.cc:492] T 0d4e1d8d235d4dc9a17a1c9e1b99c89f P e2ad591cdb7545e3abaae800c3e9aeda: No bootstrap required, opened a new log
I20260928 06:35:04.606739 28778 ts_tablet_manager.cc:1397] T 0d4e1d8d235d4dc9a17a1c9e1b99c89f P e2ad591cdb7545e3abaae800c3e9aeda: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
Sep 28 06:35:04 dist-test-slave-rrj5 krb5kdc[27796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577304, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.126@KRBTEST.COM for HTTP/127.16.157.84@KRBTEST.COM
I20260928 06:35:04.610126 28778 raft_consensus.cc:348] T 0d4e1d8d235d4dc9a17a1c9e1b99c89f P e2ad591cdb7545e3abaae800c3e9aeda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d84a82f440d4532b362d4b985579b03" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 40087 } } peers { permanent_uuid: "e2ad591cdb7545e3abaae800c3e9aeda" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39621 } } peers { permanent_uuid: "b036ed2625554b3ba52c0a9e27a5fc4e" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 33807 } }
I20260928 06:35:04.610318 28778 raft_consensus.cc:374] T 0d4e1d8d235d4dc9a17a1c9e1b99c89f P e2ad591cdb7545e3abaae800c3e9aeda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:04.610365 28778 raft_consensus.cc:729] T 0d4e1d8d235d4dc9a17a1c9e1b99c89f P e2ad591cdb7545e3abaae800c3e9aeda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2ad591cdb7545e3abaae800c3e9aeda, State: Initialized, Role: FOLLOWER
I20260928 06:35:04.610491 28781 tablet_bootstrap.cc:492] T 0d4e1d8d235d4dc9a17a1c9e1b99c89f P b036ed2625554b3ba52c0a9e27a5fc4e: Bootstrap starting.
I20260928 06:35:04.610514 28778 consensus_queue.cc:260] T 0d4e1d8d235d4dc9a17a1c9e1b99c89f P e2ad591cdb7545e3abaae800c3e9aeda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d84a82f440d4532b362d4b985579b03" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 40087 } } peers { permanent_uuid: "e2ad591cdb7545e3abaae800c3e9aeda" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39621 } } peers { permanent_uuid: "b036ed2625554b3ba52c0a9e27a5fc4e" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 33807 } }
I20260928 06:35:04.610780 28778 ts_tablet_manager.cc:1428] T 0d4e1d8d235d4dc9a17a1c9e1b99c89f P e2ad591cdb7545e3abaae800c3e9aeda: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260928 06:35:04.610901 28759 heartbeater.cc:503] Master 127.16.157.126:42009 was elected leader, sending a full tablet report...
I20260928 06:35:04.611363 28781 tablet_bootstrap.cc:654] T 0d4e1d8d235d4dc9a17a1c9e1b99c89f P b036ed2625554b3ba52c0a9e27a5fc4e: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:04.611730 28781 log.cc:824] T 0d4e1d8d235d4dc9a17a1c9e1b99c89f P b036ed2625554b3ba52c0a9e27a5fc4e: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:04.612340 28777 tablet_bootstrap.cc:492] T 0d4e1d8d235d4dc9a17a1c9e1b99c89f P 0d84a82f440d4532b362d4b985579b03: No bootstrap required, opened a new log
I20260928 06:35:04.612408 28777 ts_tablet_manager.cc:1397] T 0d4e1d8d235d4dc9a17a1c9e1b99c89f P 0d84a82f440d4532b362d4b985579b03: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260928 06:35:04.612483 28781 tablet_bootstrap.cc:492] T 0d4e1d8d235d4dc9a17a1c9e1b99c89f P b036ed2625554b3ba52c0a9e27a5fc4e: No bootstrap required, opened a new log
I20260928 06:35:04.612538 28781 ts_tablet_manager.cc:1397] T 0d4e1d8d235d4dc9a17a1c9e1b99c89f P b036ed2625554b3ba52c0a9e27a5fc4e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:35:04.614552 28777 raft_consensus.cc:348] T 0d4e1d8d235d4dc9a17a1c9e1b99c89f P 0d84a82f440d4532b362d4b985579b03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d84a82f440d4532b362d4b985579b03" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 40087 } } peers { permanent_uuid: "e2ad591cdb7545e3abaae800c3e9aeda" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39621 } } peers { permanent_uuid: "b036ed2625554b3ba52c0a9e27a5fc4e" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 33807 } }
I20260928 06:35:04.614717 28777 raft_consensus.cc:374] T 0d4e1d8d235d4dc9a17a1c9e1b99c89f P 0d84a82f440d4532b362d4b985579b03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:04.614754 28777 raft_consensus.cc:729] T 0d4e1d8d235d4dc9a17a1c9e1b99c89f P 0d84a82f440d4532b362d4b985579b03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d84a82f440d4532b362d4b985579b03, State: Initialized, Role: FOLLOWER
I20260928 06:35:04.614883 28777 consensus_queue.cc:260] T 0d4e1d8d235d4dc9a17a1c9e1b99c89f P 0d84a82f440d4532b362d4b985579b03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d84a82f440d4532b362d4b985579b03" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 40087 } } peers { permanent_uuid: "e2ad591cdb7545e3abaae800c3e9aeda" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39621 } } peers { permanent_uuid: "b036ed2625554b3ba52c0a9e27a5fc4e" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 33807 } }
I20260928 06:35:04.614966 28781 raft_consensus.cc:348] T 0d4e1d8d235d4dc9a17a1c9e1b99c89f P b036ed2625554b3ba52c0a9e27a5fc4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d84a82f440d4532b362d4b985579b03" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 40087 } } peers { permanent_uuid: "e2ad591cdb7545e3abaae800c3e9aeda" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39621 } } peers { permanent_uuid: "b036ed2625554b3ba52c0a9e27a5fc4e" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 33807 } }
I20260928 06:35:04.615120 28781 raft_consensus.cc:374] T 0d4e1d8d235d4dc9a17a1c9e1b99c89f P b036ed2625554b3ba52c0a9e27a5fc4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:04.615126 28777 ts_tablet_manager.cc:1428] T 0d4e1d8d235d4dc9a17a1c9e1b99c89f P 0d84a82f440d4532b362d4b985579b03: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:35:04.615149 28781 raft_consensus.cc:729] T 0d4e1d8d235d4dc9a17a1c9e1b99c89f P b036ed2625554b3ba52c0a9e27a5fc4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b036ed2625554b3ba52c0a9e27a5fc4e, State: Initialized, Role: FOLLOWER
I20260928 06:35:04.615257 28625 heartbeater.cc:503] Master 127.16.157.126:42009 was elected leader, sending a full tablet report...
I20260928 06:35:04.615250 28781 consensus_queue.cc:260] T 0d4e1d8d235d4dc9a17a1c9e1b99c89f P b036ed2625554b3ba52c0a9e27a5fc4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d84a82f440d4532b362d4b985579b03" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 40087 } } peers { permanent_uuid: "e2ad591cdb7545e3abaae800c3e9aeda" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39621 } } peers { permanent_uuid: "b036ed2625554b3ba52c0a9e27a5fc4e" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 33807 } }
I20260928 06:35:04.615478 28781 ts_tablet_manager.cc:1428] T 0d4e1d8d235d4dc9a17a1c9e1b99c89f P b036ed2625554b3ba52c0a9e27a5fc4e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:35:04.615757 28487 heartbeater.cc:503] Master 127.16.157.126:42009 was elected leader, sending a full tablet report...
I20260928 06:35:04.626511 28782 raft_consensus.cc:482] T 0d4e1d8d235d4dc9a17a1c9e1b99c89f P e2ad591cdb7545e3abaae800c3e9aeda [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:35:04.626644 28782 raft_consensus.cc:504] T 0d4e1d8d235d4dc9a17a1c9e1b99c89f P e2ad591cdb7545e3abaae800c3e9aeda [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d84a82f440d4532b362d4b985579b03" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 40087 } } peers { permanent_uuid: "e2ad591cdb7545e3abaae800c3e9aeda" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39621 } } peers { permanent_uuid: "b036ed2625554b3ba52c0a9e27a5fc4e" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 33807 } }
I20260928 06:35:04.627537 28782 leader_election.cc:288] T 0d4e1d8d235d4dc9a17a1c9e1b99c89f P e2ad591cdb7545e3abaae800c3e9aeda [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d84a82f440d4532b362d4b985579b03 (127.16.157.66:40087), b036ed2625554b3ba52c0a9e27a5fc4e (127.16.157.65:33807)
I20260928 06:35:04.630784 28761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:04.627157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.66:40087 (local address 127.16.157.67:58701)
0928 06:35:04.627208 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:04.627222 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:04.627275 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:04.627638 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:04.627639 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:04.627640 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:04.627709 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:04.627710 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:04.628624 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:04.628625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:04.629754 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:04.629757 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:04.630540 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:04.630541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:04.630621 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:04.630625 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:35:04.630672 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 06:35:04.630712 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260928 06:35:04.631079 28772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:04.627263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:58701 (local address 127.16.157.66:40087)
0928 06:35:04.627356 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:04.627358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:04.627366 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:35:04.627389 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:04.627390 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:04.627401 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:04.627446 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:04.627808 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:04.628484 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:04.629870 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:04.630425 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:04.630499 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:04.630508 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:04.630967 (+ 459us) server_negotiation.cc:299] Negotiation successful
0928 06:35:04.631018 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 06:35:04.631357 28563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d4e1d8d235d4dc9a17a1c9e1b99c89f" candidate_uuid: "e2ad591cdb7545e3abaae800c3e9aeda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d84a82f440d4532b362d4b985579b03" is_pre_election: true
I20260928 06:35:04.631510 28563 raft_consensus.cc:2454] T 0d4e1d8d235d4dc9a17a1c9e1b99c89f P 0d84a82f440d4532b362d4b985579b03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2ad591cdb7545e3abaae800c3e9aeda in term 0.
I20260928 06:35:04.631840 28647 leader_election.cc:302] T 0d4e1d8d235d4dc9a17a1c9e1b99c89f P e2ad591cdb7545e3abaae800c3e9aeda [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d84a82f440d4532b362d4b985579b03, e2ad591cdb7545e3abaae800c3e9aeda; no voters:
I20260928 06:35:04.632037 28782 raft_consensus.cc:2781] T 0d4e1d8d235d4dc9a17a1c9e1b99c89f P e2ad591cdb7545e3abaae800c3e9aeda [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:35:04.632093 28782 raft_consensus.cc:482] T 0d4e1d8d235d4dc9a17a1c9e1b99c89f P e2ad591cdb7545e3abaae800c3e9aeda [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:35:04.632117 28782 raft_consensus.cc:3037] T 0d4e1d8d235d4dc9a17a1c9e1b99c89f P e2ad591cdb7545e3abaae800c3e9aeda [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:04.633085 28782 raft_consensus.cc:504] T 0d4e1d8d235d4dc9a17a1c9e1b99c89f P e2ad591cdb7545e3abaae800c3e9aeda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d84a82f440d4532b362d4b985579b03" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 40087 } } peers { permanent_uuid: "e2ad591cdb7545e3abaae800c3e9aeda" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39621 } } peers { permanent_uuid: "b036ed2625554b3ba52c0a9e27a5fc4e" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 33807 } }
I20260928 06:35:04.633183 28776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:04.627573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:50913 (local address 127.16.157.65:33807)
0928 06:35:04.627608 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:04.627610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:04.628793 (+ 1183us) server_negotiation.cc:407] Connection header received
0928 06:35:04.628889 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:04.628890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:04.628900 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:04.628928 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:04.629937 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:04.630561 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:04.632124 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:04.632758 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:04.632830 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:04.632839 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:04.633078 (+ 239us) server_negotiation.cc:299] Negotiation successful
0928 06:35:04.633120 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:35:04.633212 28782 leader_election.cc:288] T 0d4e1d8d235d4dc9a17a1c9e1b99c89f P e2ad591cdb7545e3abaae800c3e9aeda [CANDIDATE]: Term 1 election: Requested vote from peers 0d84a82f440d4532b362d4b985579b03 (127.16.157.66:40087), b036ed2625554b3ba52c0a9e27a5fc4e (127.16.157.65:33807)
I20260928 06:35:04.633467 28786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:04.627412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.65:33807 (local address 127.16.157.67:50913)
0928 06:35:04.628725 (+ 1313us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:04.628736 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:04.628819 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:04.628986 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:04.628987 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:04.628988 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:04.629056 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:04.629057 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:04.630660 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:04.630662 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:04.631998 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:04.632001 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:04.632895 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:04.632896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:04.632984 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:04.632987 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:35:04.633309 (+ 322us) client_negotiation.cc:240] Negotiation successful
0928 06:35:04.633356 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1294,"thread_start_us":92,"threads_started":1}
I20260928 06:35:04.633647 28563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d4e1d8d235d4dc9a17a1c9e1b99c89f" candidate_uuid: "e2ad591cdb7545e3abaae800c3e9aeda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d84a82f440d4532b362d4b985579b03"
I20260928 06:35:04.633726 28563 raft_consensus.cc:3037] T 0d4e1d8d235d4dc9a17a1c9e1b99c89f P 0d84a82f440d4532b362d4b985579b03 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:04.633791 28431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d4e1d8d235d4dc9a17a1c9e1b99c89f" candidate_uuid: "e2ad591cdb7545e3abaae800c3e9aeda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b036ed2625554b3ba52c0a9e27a5fc4e" is_pre_election: true
I20260928 06:35:04.633937 28431 raft_consensus.cc:2454] T 0d4e1d8d235d4dc9a17a1c9e1b99c89f P b036ed2625554b3ba52c0a9e27a5fc4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2ad591cdb7545e3abaae800c3e9aeda in term 0.
I20260928 06:35:04.634069 28436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d4e1d8d235d4dc9a17a1c9e1b99c89f" candidate_uuid: "e2ad591cdb7545e3abaae800c3e9aeda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b036ed2625554b3ba52c0a9e27a5fc4e"
I20260928 06:35:04.634128 28436 raft_consensus.cc:3037] T 0d4e1d8d235d4dc9a17a1c9e1b99c89f P b036ed2625554b3ba52c0a9e27a5fc4e [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:04.634656 28563 raft_consensus.cc:2454] T 0d4e1d8d235d4dc9a17a1c9e1b99c89f P 0d84a82f440d4532b362d4b985579b03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2ad591cdb7545e3abaae800c3e9aeda in term 1.
I20260928 06:35:04.634881 28647 leader_election.cc:302] T 0d4e1d8d235d4dc9a17a1c9e1b99c89f P e2ad591cdb7545e3abaae800c3e9aeda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d84a82f440d4532b362d4b985579b03, e2ad591cdb7545e3abaae800c3e9aeda; no voters:
I20260928 06:35:04.635038 28782 raft_consensus.cc:2781] T 0d4e1d8d235d4dc9a17a1c9e1b99c89f P e2ad591cdb7545e3abaae800c3e9aeda [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:04.635054 28436 raft_consensus.cc:2454] T 0d4e1d8d235d4dc9a17a1c9e1b99c89f P b036ed2625554b3ba52c0a9e27a5fc4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2ad591cdb7545e3abaae800c3e9aeda in term 1.
I20260928 06:35:04.635226 28782 raft_consensus.cc:686] T 0d4e1d8d235d4dc9a17a1c9e1b99c89f P e2ad591cdb7545e3abaae800c3e9aeda [term 1 LEADER]: Becoming Leader. State: Replica: e2ad591cdb7545e3abaae800c3e9aeda, State: Running, Role: LEADER
I20260928 06:35:04.635313 28782 consensus_queue.cc:237] T 0d4e1d8d235d4dc9a17a1c9e1b99c89f P e2ad591cdb7545e3abaae800c3e9aeda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d84a82f440d4532b362d4b985579b03" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 40087 } } peers { permanent_uuid: "e2ad591cdb7545e3abaae800c3e9aeda" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39621 } } peers { permanent_uuid: "b036ed2625554b3ba52c0a9e27a5fc4e" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 33807 } }
I20260928 06:35:04.636206 28275 catalog_manager.cc:5495] T 0d4e1d8d235d4dc9a17a1c9e1b99c89f P e2ad591cdb7545e3abaae800c3e9aeda reported cstate change: term changed from 0 to 1, leader changed from <none> to e2ad591cdb7545e3abaae800c3e9aeda (127.16.157.67). New cstate: current_term: 1 leader_uuid: "e2ad591cdb7545e3abaae800c3e9aeda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d84a82f440d4532b362d4b985579b03" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 40087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2ad591cdb7545e3abaae800c3e9aeda" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39621 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b036ed2625554b3ba52c0a9e27a5fc4e" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 33807 } health_report { overall_health: UNKNOWN } } }
I20260928 06:35:04.654919 28774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:04.651359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40436 (local address 127.16.157.67:39621)
0928 06:35:04.651401 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:04.651403 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:04.651410 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:35:04.651447 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:04.651448 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:04.651467 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:04.651528 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:35:04.651736 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:04.652475 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:04.653674 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:04.653994 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:04.654060 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:04.654271 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:35:04.654414 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:35:04.654492 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:04.654816 (+ 324us) server_negotiation.cc:299] Negotiation successful
0928 06:35:04.654839 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 06:35:04.657127 28436 raft_consensus.cc:1261] T 0d4e1d8d235d4dc9a17a1c9e1b99c89f P b036ed2625554b3ba52c0a9e27a5fc4e [term 1 FOLLOWER]: Refusing update from remote peer e2ad591cdb7545e3abaae800c3e9aeda: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:35:04.657469 28787 consensus_queue.cc:1035] T 0d4e1d8d235d4dc9a17a1c9e1b99c89f P e2ad591cdb7545e3abaae800c3e9aeda [LEADER]: Connected to new peer: Peer: permanent_uuid: "b036ed2625554b3ba52c0a9e27a5fc4e" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:35:04.657898 28563 raft_consensus.cc:1261] T 0d4e1d8d235d4dc9a17a1c9e1b99c89f P 0d84a82f440d4532b362d4b985579b03 [term 1 FOLLOWER]: Refusing update from remote peer e2ad591cdb7545e3abaae800c3e9aeda: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:35:04.659157 28787 consensus_queue.cc:1035] T 0d4e1d8d235d4dc9a17a1c9e1b99c89f P e2ad591cdb7545e3abaae800c3e9aeda [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d84a82f440d4532b362d4b985579b03" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 40087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:35:04.663757 28790 mvcc.cc:205] Tried to move back new op lower bound from 7334204639872278528 to 7334204639786852352. Current Snapshot: MvccSnapshot[applied={T|T < 7334204639872278528}]
I20260928 06:35:04.664278 28793 mvcc.cc:205] Tried to move back new op lower bound from 7334204639872278528 to 7334204639786852352. Current Snapshot: MvccSnapshot[applied={T|T < 7334204639872278528}]
I20260928 06:35:04.666685 28791 mvcc.cc:205] Tried to move back new op lower bound from 7334204639872278528 to 7334204639786852352. Current Snapshot: MvccSnapshot[applied={T|T < 7334204639872278528}]
I20260928 06:35:04.680622 28275 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:35:04.680744 28275 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:35:04.681749 28275 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7e7a05cb5d36487bb01a925538c66b16: Sending DeleteTablet for 3 replicas of tablet 0d4e1d8d235d4dc9a17a1c9e1b99c89f
I20260928 06:35:04.682298 28559 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d4e1d8d235d4dc9a17a1c9e1b99c89f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:04 UTC) from {username='kudu', principal='kudu/127.16.157.126@KRBTEST.COM'} at 127.0.0.1:46170
I20260928 06:35:04.682543 28421 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d4e1d8d235d4dc9a17a1c9e1b99c89f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:04 UTC) from {username='kudu', principal='kudu/127.16.157.126@KRBTEST.COM'} at 127.0.0.1:35798
I20260928 06:35:04.683161 28693 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d4e1d8d235d4dc9a17a1c9e1b99c89f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:04 UTC) from {username='kudu', principal='kudu/127.16.157.126@KRBTEST.COM'} at 127.0.0.1:40420
I20260928 06:35:04.683341 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 28356
I20260928 06:35:04.693689 28799 tablet_replica.cc:330] T 0d4e1d8d235d4dc9a17a1c9e1b99c89f P 0d84a82f440d4532b362d4b985579b03: stopping tablet replica
I20260928 06:35:04.693921 28799 raft_consensus.cc:2229] T 0d4e1d8d235d4dc9a17a1c9e1b99c89f P 0d84a82f440d4532b362d4b985579b03 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:35:04.694157 28799 raft_consensus.cc:2258] T 0d4e1d8d235d4dc9a17a1c9e1b99c89f P 0d84a82f440d4532b362d4b985579b03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:04.698794 28799 ts_tablet_manager.cc:1905] T 0d4e1d8d235d4dc9a17a1c9e1b99c89f P 0d84a82f440d4532b362d4b985579b03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:04.700525 28801 tablet_replica.cc:330] T 0d4e1d8d235d4dc9a17a1c9e1b99c89f P e2ad591cdb7545e3abaae800c3e9aeda: stopping tablet replica
I20260928 06:35:04.700703 28801 raft_consensus.cc:2229] T 0d4e1d8d235d4dc9a17a1c9e1b99c89f P e2ad591cdb7545e3abaae800c3e9aeda [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:35:04.700928 28801 raft_consensus.cc:2258] T 0d4e1d8d235d4dc9a17a1c9e1b99c89f P e2ad591cdb7545e3abaae800c3e9aeda [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:04.701149 28799 ts_tablet_manager.cc:1918] T 0d4e1d8d235d4dc9a17a1c9e1b99c89f P 0d84a82f440d4532b362d4b985579b03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:35:04.701593 28801 ts_tablet_manager.cc:1905] T 0d4e1d8d235d4dc9a17a1c9e1b99c89f P e2ad591cdb7545e3abaae800c3e9aeda: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:04.701725 28799 log.cc:1196] T 0d4e1d8d235d4dc9a17a1c9e1b99c89f P 0d84a82f440d4532b362d4b985579b03: Deleting WAL directory at /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/ts-1/wal/wals/0d4e1d8d235d4dc9a17a1c9e1b99c89f
I20260928 06:35:04.702384 28799 ts_tablet_manager.cc:1939] T 0d4e1d8d235d4dc9a17a1c9e1b99c89f P 0d84a82f440d4532b362d4b985579b03: Deleting consensus metadata
I20260928 06:35:04.703908 28801 ts_tablet_manager.cc:1918] T 0d4e1d8d235d4dc9a17a1c9e1b99c89f P e2ad591cdb7545e3abaae800c3e9aeda: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:35:04.704459 28260 catalog_manager.cc:4841] TS 0d84a82f440d4532b362d4b985579b03 (127.16.157.66:40087): tablet 0d4e1d8d235d4dc9a17a1c9e1b99c89f (table test-table [id=76f0861164244a9895a136fe8670c3ab]) successfully deleted
I20260928 06:35:04.704662 28801 log.cc:1196] T 0d4e1d8d235d4dc9a17a1c9e1b99c89f P e2ad591cdb7545e3abaae800c3e9aeda: Deleting WAL directory at /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577196946045-17013-0/minicluster-data/ts-2/wal/wals/0d4e1d8d235d4dc9a17a1c9e1b99c89f
I20260928 06:35:04.704998 28801 ts_tablet_manager.cc:1939] T 0d4e1d8d235d4dc9a17a1c9e1b99c89f P e2ad591cdb7545e3abaae800c3e9aeda: Deleting consensus metadata
W20260928 06:35:04.705294 28261 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.157.65:33807 (error 108)
I20260928 06:35:04.705915 28771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:04.705761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.65:33807 (local address 127.0.0.1:35804)
0928 06:35:04.705827 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:04.705878 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.65:33807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 06:35:04.706050 28261 catalog_manager.cc:4584] TS b036ed2625554b3ba52c0a9e27a5fc4e (127.16.157.65:33807): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0d4e1d8d235d4dc9a17a1c9e1b99c89f: Network error: Client connection negotiation failed: client connection to 127.16.157.65:33807: connect: Connection refused (error 111)
I20260928 06:35:04.706431 28259 catalog_manager.cc:4841] TS e2ad591cdb7545e3abaae800c3e9aeda (127.16.157.67:39621): tablet 0d4e1d8d235d4dc9a17a1c9e1b99c89f (table test-table [id=76f0861164244a9895a136fe8670c3ab]) successfully deleted
I20260928 06:35:04.706677 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 28495
I20260928 06:35:04.713068 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 28630
I20260928 06:35:04.718827 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 28243
I20260928 06:35:04.724166 17013 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 06:35:05.068221 17013 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 06:35:05.068347 17013 mini_ranger.cc:66] Stopping Ranger...
I20260928 06:35:05.413275 17013 mini_ranger.cc:68] Stopped Ranger
2026-09-28 06:35:05.413 UTC [27826] LOG: received smart shutdown request
2026-09-28 06:35:05.416 UTC [27826] LOG: background worker "logical replication launcher" (PID 27835) exited with exit code 1
2026-09-28 06:35:05.417 UTC [27830] LOG: shutting down
2026-09-28 06:35:05.570 UTC [27826] LOG: database system is shut down
2026-09-28T06:35:05Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (51911 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:35:05 dist-test-slave-rrj5 krb5kdc[28811](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:35:05 dist-test-slave-rrj5 krb5kdc[28811](info): set up 2 sockets
Sep 28 06:35:05 dist-test-slave-rrj5 krb5kdc[28811](info): commencing operation
krb5kdc: starting...
W20260928 06:35:06.758158 17013 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Sep 28 06:35:06 dist-test-slave-rrj5 krb5kdc[28811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:35:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:35:06Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.16.157.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.16.157.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.16.157.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/krb5kdc/rangeradmin_127.16.157.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.16.157.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/krb5kdc/rangeradmin_127.16.157.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.16.157.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.16.157.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.16.157.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/krb5kdc/rangerlookup_127.16.157.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.16.157.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/krb5kdc/rangerlookup_127.16.157.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.157.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.16.157.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.157.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/krb5kdc/HTTP_127.16.157.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.157.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/krb5kdc/HTTP_127.16.157.84@KRBTEST.COM.keytab.
I20260928 06:35:06.926549 17013 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskGDaHLl/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/postgres -l logfile start
2026-09-28 06:35:09.334 UTC [28841] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 06:35:09.334 UTC [28841] LOG: listening on IPv4 address "127.16.157.84", port 52387
2026-09-28 06:35:09.337 UTC [28841] LOG: listening on Unix socket "/tmp/.s.PGSQL.52387"
2026-09-28 06:35:09.372 UTC [28844] LOG: database system was shut down at 2026-09-28 06:35:07 UTC
2026-09-28 06:35:09.376 UTC [28841] LOG: database system is ready to accept connections
I20260928 06:35:10.314159 17013 mini_postgres.cc:96] Postgres bound to 52387
2026-09-28 06:35:10.321 UTC [28852] FATAL: database "slave" does not exist
127.16.157.84:52387 - accepting connections
I20260928 06:35:10.322178 17013 mini_ranger.cc:161] Starting Ranger...
I20260928 06:35:10.336738 17013 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 06:35:10.757669 17013 mini_ranger.cc:87] Created ranger Postgres database
I20260928 06:35:10.757735 17013 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ranger-admin
2026-09-28 06:35:11,042 [I] DB FLAVOR :POSTGRES
2026-09-28 06:35:11,042 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 06:35:11,043 [I] Checking connection..
2026-09-28 06:35:11,043 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGDaHLl/build/release/bin/postgresql.jar:/tmp/dist-test-taskGDaHLl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.157.84:52387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 06:35:11,272 [I] Checking connection passed.
2026-09-28 06:35:11,273 [I] --------- Verifying version history table ---------
2026-09-28 06:35:11,273 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGDaHLl/build/release/bin/postgresql.jar:/tmp/dist-test-taskGDaHLl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.157.84:52387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 06:35:11,513 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 06:35:11,514 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGDaHLl/build/release/bin/postgresql.jar:/tmp/dist-test-taskGDaHLl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.157.84:52387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 06:35:11,770 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 06:35:11,770 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 06:35:11,770 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGDaHLl/build/release/bin/postgresql.jar:/tmp/dist-test-taskGDaHLl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.157.84:52387/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGDaHLl/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 06:35:12.009 UTC [28916] WARNING: there is no transaction in progress
2026-09-28 06:35:12,015 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 06:35:12,015 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGDaHLl/build/release/bin/postgresql.jar:/tmp/dist-test-taskGDaHLl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.157.84:52387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 06:35:12,241 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 06:35:12,241 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 06:35:12,241 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGDaHLl/build/release/bin/postgresql.jar:/tmp/dist-test-taskGDaHLl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.157.84:52387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 06:35:12,487 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGDaHLl/build/release/bin/postgresql.jar:/tmp/dist-test-taskGDaHLl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.157.84:52387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 06:35:12,730 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGDaHLl/build/release/bin/postgresql.jar:/tmp/dist-test-taskGDaHLl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.157.84:52387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-rrj5.c.gcp-upstream.internal','N') ;"
2026-09-28 06:35:12,979 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGDaHLl/build/release/bin/postgresql.jar:/tmp/dist-test-taskGDaHLl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.157.84:52387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 06:35:13,286 [I] Table x_portal_user does not exist in database ranger
2026-09-28 06:35:13,286 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGDaHLl/build/release/bin/postgresql.jar:/tmp/dist-test-taskGDaHLl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.157.84:52387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 06:35:13,520 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 06:35:13,520 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGDaHLl/build/release/bin/postgresql.jar:/tmp/dist-test-taskGDaHLl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.157.84:52387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 06:35:13,775 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGDaHLl/build/release/bin/postgresql.jar:/tmp/dist-test-taskGDaHLl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.157.84:52387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 06:35:14,018 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 06:35:14,018 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGDaHLl/build/release/bin/postgresql.jar:/tmp/dist-test-taskGDaHLl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.157.84:52387/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGDaHLl/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 06:35:14.966 UTC [29024] WARNING: there is no transaction in progress
2026-09-28 06:35:14.985 UTC [29024] WARNING: there is no transaction in progress
2026-09-28 06:35:15.001 UTC [29024] WARNING: there is no transaction in progress
2026-09-28 06:35:15.015 UTC [29024] WARNING: there is no transaction in progress
2026-09-28 06:35:15.031 UTC [29024] WARNING: there is no transaction in progress
2026-09-28 06:35:15.049 UTC [29024] WARNING: there is no transaction in progress
2026-09-28 06:35:15.105 UTC [29024] WARNING: there is no transaction in progress
2026-09-28 06:35:15.111 UTC [29024] WARNING: there is no transaction in progress
2026-09-28 06:35:15.125 UTC [29024] WARNING: there is no transaction in progress
2026-09-28 06:35:15.134 UTC [29024] WARNING: there is no transaction in progress
2026-09-28 06:35:15.144 UTC [29024] WARNING: there is no transaction in progress
2026-09-28 06:35:15.157 UTC [29024] WARNING: there is no transaction in progress
2026-09-28 06:35:15.166 UTC [29024] WARNING: there is no transaction in progress
2026-09-28 06:35:15.172 UTC [29024] WARNING: there is no transaction in progress
2026-09-28 06:35:15.181 UTC [29024] WARNING: there is no transaction in progress
2026-09-28 06:35:15.188 UTC [29024] WARNING: there is no transaction in progress
2026-09-28 06:35:15,817 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 06:35:15,817 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGDaHLl/build/release/bin/postgresql.jar:/tmp/dist-test-taskGDaHLl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.157.84:52387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 06:35:16,048 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 06:35:16,049 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGDaHLl/build/release/bin/postgresql.jar:/tmp/dist-test-taskGDaHLl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.157.84:52387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 06:35:16,283 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 06:35:16,283 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGDaHLl/build/release/bin/postgresql.jar:/tmp/dist-test-taskGDaHLl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.157.84:52387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 06:35:16,529 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 06:35:16,530 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGDaHLl/build/release/bin/postgresql.jar:/tmp/dist-test-taskGDaHLl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.157.84:52387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 06:35:16,795 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGDaHLl/build/release/bin/postgresql.jar:/tmp/dist-test-taskGDaHLl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.157.84:52387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 06:35:17,047 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGDaHLl/build/release/bin/postgresql.jar:/tmp/dist-test-taskGDaHLl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.157.84:52387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-rrj5.c.gcp-upstream.internal';"
2026-09-28 06:35:17,292 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 06:35:17,292 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGDaHLl/build/release/bin/postgresql.jar:/tmp/dist-test-taskGDaHLl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.157.84:52387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 06:35:17,546 [I] DB_PATCHES have already been applied
I20260928 06:35:17.551820 17013 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ranger-admin:/tmp/dist-test-taskGDaHLl/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskGDaHLl/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskGDaHLl/build/release/bin/postgresql.jar:/tmp/dist-test-taskGDaHLl/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 06:35:17.551890 17013 mini_ranger.cc:193] Using host: 127.16.157.84
I20260928 06:35:17.555136 17013 mini_ranger.cc:227] Ranger admin URL: http://127.16.157.84:33525
Sep 28, 2026 6:35:17 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 6:35:17 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 6:35:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 6:35:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 6:35:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 6:35:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 6:35:18 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 06:35:18.643015 17013 mini_ranger.cc:160] Time spent starting Ranger: real 8.321s user 0.000s sys 0.009s
Sep 28, 2026 6:35:38 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 6:35:38 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 6:35:38 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 6:35:38 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 6:35:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 6:35:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 6:35:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 6:35:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 6:35:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 6:35:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 6:35:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 6:35:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 6:35:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 6:35:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 6:35:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 6:35:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 6:35:38 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 6:35:39 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 06:35:39.766784 17013 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.16.157.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.16.157.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.16.157.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/krb5kdc/rangerkms_127.16.157.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.16.157.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/krb5kdc/rangerkms_127.16.157.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.157.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.16.157.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.157.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/krb5kdc/HTTP_127.16.157.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.157.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/krb5kdc/HTTP_127.16.157.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 06:35:39 dist-test-slave-rrj5 krb5kdc[28811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577339, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 06:35:39.863756 17013 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 06:35:39.872721 17013 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 06:35:40.700084 17013 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 06:35:40.700167 17013 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ranger-kms
I20260928 06:35:40.700186 17013 mini_ranger_kms.cc:227] Using postgres at 127.16.157.84:52387
2026-09-28 06:35:40,869 [I] DB FLAVOR :POSTGRES
2026-09-28 06:35:40,870 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 06:35:40,870 [I] Checking connection
2026-09-28 06:35:40,870 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGDaHLl/build/release/bin/postgresql.jar:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.157.84:52387/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 06:35:41,104 [I] connection success
2026-09-28 06:35:41,104 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 06:35:41,104 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 06:35:41,104 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGDaHLl/build/release/bin/postgresql.jar:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.157.84:52387/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 06:35:41,332 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 06:35:41,333 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 06:35:41,333 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 06:35:41,333 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGDaHLl/build/release/bin/postgresql.jar:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.157.84:52387/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 06:35:41,604 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 06:35:42.053552 17013 mini_ranger_kms.cc:310] Created kms service
I20260928 06:35:42.237715 17013 mini_ranger_kms.cc:326] Created kudu user
I20260928 06:35:42.279448 17013 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 06:35:42 dist-test-slave-rrj5 krb5kdc[28811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577339, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.16.157.84@KRBTEST.COM
I20260928 06:35:42.648581 17013 mini_ranger_kms.cc:384] Added ranger policy
I20260928 06:35:42.648769 17013 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ranger-kms:/tmp/dist-test-taskGDaHLl/build/release/bin/postgresql.jar:/tmp/dist-test-taskGDaHLl/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskGDaHLl/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 06:35:42.648792 17013 mini_ranger_kms.cc:240] Using host: 127.16.157.84
I20260928 06:35:42.652133 17013 mini_ranger_kms.cc:276] Ranger KMS PID: 29204
I20260928 06:35:42.652230 17013 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.16.157.84:58421
Sep 28, 2026 6:35:43 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 6:35:43 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 6:35:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 6:35:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 6:35:44 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 6:35:44 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 28 06:35:44 dist-test-slave-rrj5 krb5kdc[28811](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790577344, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.157.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:44.907014 17013 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.043s user 0.000s sys 0.153s
I20260928 06:35:44.907120 17013 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 06:35:44.907184 17013 mini_ranger_kms.cc:401] 127.16.157.84:58421/kms/v1/keys
Sep 28 06:35:45 dist-test-slave-rrj5 krb5kdc[28811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577344, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.157.84@KRBTEST.COM for HTTP/127.16.157.84@KRBTEST.COM
Sep 28 06:35:45 dist-test-slave-rrj5 krb5kdc[28811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577344, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.157.84@KRBTEST.COM for HTTP/127.16.157.84@KRBTEST.COM
Sep 28 06:35:47 dist-test-slave-rrj5 krb5kdc[28811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.16.157.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab.
Entry for principal kudu/127.16.157.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab.
I20260928 06:35:47.972502 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:44457
--webserver_interface=127.16.157.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=kudu/127.16.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:35393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.157.126:44457
--ranger_config_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.157.84:58421/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:48.053527 29255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:48.053856 29255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:48.053880 29255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:48.055080 29255 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:48.055145 29255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:48.055190 29255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:48.055222 29255 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:48.055243 29255 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:35:48.056815 29255 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:35393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/master-0/wal
--ranger_kms_url=127.16.157.84:58421/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.16.157.126:44457
--ranger_config_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=kudu/127.16.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:44457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:35:48.057060 29255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:48.057464 29255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:48.062647 29263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:48.062731 29261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:48.062808 29260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:48.062734 29255 server_base.cc:1034] running on GCE node
I20260928 06:35:48.063241 29255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:48.063580 29255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:48.064754 29255 hybrid_clock.cc:648] HybridClock initialized: now 1790577348064728 us; error 42 us; skew 500 ppm
Sep 28 06:35:48 dist-test-slave-rrj5 krb5kdc[28811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577348, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:48.067641 29255 init.cc:377] Logged in from keytab as kudu/127.16.157.126@KRBTEST.COM (short username kudu)
I20260928 06:35:48.068095 29255 webserver.cc:460] Webserver started at http://127.16.157.126:37573/ using document root <none> and password file <none>
I20260928 06:35:48.068315 29255 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:48.068375 29255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:48.068522 29255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 06:35:48 dist-test-slave-rrj5 krb5kdc[28811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577348, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.126@KRBTEST.COM for HTTP/127.16.157.84@KRBTEST.COM
I20260928 06:35:48.129311 29255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/master-0/data/instance:
uuid: "b26988d78de3439d8a7817a65dc7b1f6"
format_stamp: "Formatted at 2026-09-28 06:35:48 on dist-test-slave-rrj5"
server_key: "42685550e9074b57de75ec2db83e6ae8"
server_key_iv: "c21a82ac816daad519a21462debc4dce"
server_key_version: "kuduclusterkey@0"
I20260928 06:35:48.129771 29255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/master-0/wal/instance:
uuid: "b26988d78de3439d8a7817a65dc7b1f6"
format_stamp: "Formatted at 2026-09-28 06:35:48 on dist-test-slave-rrj5"
server_key: "42685550e9074b57de75ec2db83e6ae8"
server_key_iv: "c21a82ac816daad519a21462debc4dce"
server_key_version: "kuduclusterkey@0"
I20260928 06:35:48.131270 29255 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:35:48.142707 29270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:48.142925 29255 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:35:48.143051 29255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/master-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/master-0/wal
uuid: "b26988d78de3439d8a7817a65dc7b1f6"
format_stamp: "Formatted at 2026-09-28 06:35:48 on dist-test-slave-rrj5"
server_key: "42685550e9074b57de75ec2db83e6ae8"
server_key_iv: "c21a82ac816daad519a21462debc4dce"
server_key_version: "kuduclusterkey@0"
I20260928 06:35:48.143170 29255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:48.167797 29255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:48.169922 29255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:48.170089 29255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:48.174971 29255 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.126:44457
I20260928 06:35:48.175060 29322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.126:44457 every 8 connection(s)
I20260928 06:35:48.175396 29255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
I20260928 06:35:48.176107 29323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:48.178231 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 29255
I20260928 06:35:48.178329 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/master-0/wal/instance
I20260928 06:35:48.178530 17013 external_mini_cluster.cc:1428] Setting key 68427f7ac32d617df45fc607921440c2
I20260928 06:35:48.178598 29323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b26988d78de3439d8a7817a65dc7b1f6: Bootstrap starting.
I20260928 06:35:48.179327 29323 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b26988d78de3439d8a7817a65dc7b1f6: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:48.179643 29323 log.cc:824] T 00000000000000000000000000000000 P b26988d78de3439d8a7817a65dc7b1f6: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:48.180466 29323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b26988d78de3439d8a7817a65dc7b1f6: No bootstrap required, opened a new log
I20260928 06:35:48.182444 29323 raft_consensus.cc:348] T 00000000000000000000000000000000 P b26988d78de3439d8a7817a65dc7b1f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b26988d78de3439d8a7817a65dc7b1f6" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 44457 } }
I20260928 06:35:48.182646 29323 raft_consensus.cc:374] T 00000000000000000000000000000000 P b26988d78de3439d8a7817a65dc7b1f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:48.182684 29323 raft_consensus.cc:729] T 00000000000000000000000000000000 P b26988d78de3439d8a7817a65dc7b1f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b26988d78de3439d8a7817a65dc7b1f6, State: Initialized, Role: FOLLOWER
I20260928 06:35:48.182829 29323 consensus_queue.cc:260] T 00000000000000000000000000000000 P b26988d78de3439d8a7817a65dc7b1f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b26988d78de3439d8a7817a65dc7b1f6" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 44457 } }
I20260928 06:35:48.182910 29323 raft_consensus.cc:388] T 00000000000000000000000000000000 P b26988d78de3439d8a7817a65dc7b1f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:35:48.182940 29323 raft_consensus.cc:482] T 00000000000000000000000000000000 P b26988d78de3439d8a7817a65dc7b1f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:35:48.182984 29323 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b26988d78de3439d8a7817a65dc7b1f6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:48.183655 29323 raft_consensus.cc:504] T 00000000000000000000000000000000 P b26988d78de3439d8a7817a65dc7b1f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b26988d78de3439d8a7817a65dc7b1f6" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 44457 } }
I20260928 06:35:48.183775 29323 leader_election.cc:302] T 00000000000000000000000000000000 P b26988d78de3439d8a7817a65dc7b1f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b26988d78de3439d8a7817a65dc7b1f6; no voters:
I20260928 06:35:48.183965 29323 leader_election.cc:288] T 00000000000000000000000000000000 P b26988d78de3439d8a7817a65dc7b1f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:35:48.183988 29328 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b26988d78de3439d8a7817a65dc7b1f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:48.184207 29328 raft_consensus.cc:686] T 00000000000000000000000000000000 P b26988d78de3439d8a7817a65dc7b1f6 [term 1 LEADER]: Becoming Leader. State: Replica: b26988d78de3439d8a7817a65dc7b1f6, State: Running, Role: LEADER
I20260928 06:35:48.184245 29323 sys_catalog.cc:564] T 00000000000000000000000000000000 P b26988d78de3439d8a7817a65dc7b1f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:35:48.184381 29328 consensus_queue.cc:237] T 00000000000000000000000000000000 P b26988d78de3439d8a7817a65dc7b1f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b26988d78de3439d8a7817a65dc7b1f6" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 44457 } }
I20260928 06:35:48.184979 29328 sys_catalog.cc:455] T 00000000000000000000000000000000 P b26988d78de3439d8a7817a65dc7b1f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b26988d78de3439d8a7817a65dc7b1f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b26988d78de3439d8a7817a65dc7b1f6" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 44457 } } }
I20260928 06:35:48.185140 29328 sys_catalog.cc:458] T 00000000000000000000000000000000 P b26988d78de3439d8a7817a65dc7b1f6 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:48.185310 29328 sys_catalog.cc:455] T 00000000000000000000000000000000 P b26988d78de3439d8a7817a65dc7b1f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b26988d78de3439d8a7817a65dc7b1f6. Latest consensus state: current_term: 1 leader_uuid: "b26988d78de3439d8a7817a65dc7b1f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b26988d78de3439d8a7817a65dc7b1f6" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 44457 } } }
I20260928 06:35:48.185433 29328 sys_catalog.cc:458] T 00000000000000000000000000000000 P b26988d78de3439d8a7817a65dc7b1f6 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:48.185609 29337 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:35:48.185775 29323 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Sep 28 06:35:48 dist-test-slave-rrj5 krb5kdc[28811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:35:48.188660 29337 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:35:48.197777 29326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:48.179042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54928 (local address 127.16.157.126:44457)
0928 06:35:48.179221 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:48.179241 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:48.179272 (+ 31us) server_negotiation.cc:407] Connection header received
0928 06:35:48.179828 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:48.179838 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:48.179914 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:48.180084 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:48.180872 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:48.182384 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:48.183516 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:48.184141 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:48.187818 (+ 3677us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:48.188406 (+ 588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:48.188414 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:48.188414 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:48.188425 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:48.195474 (+ 7049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:48.195979 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:48.195982 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:48.195982 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:48.196096 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:48.196346 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:48.196347 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:48.196347 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:48.196724 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:48.197340 (+ 616us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:48.197394 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 06:35:48.197503 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":70,"threads_started":1}
I20260928 06:35:48.201231 29337 catalog_manager.cc:1347] Generated new cluster ID: 2b9a40c0a075474c8e1803f6ebad16ff
I20260928 06:35:48.201344 29337 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:35:48.206328 29337 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:35:48.207099 29337 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:35:48.215529 29337 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b26988d78de3439d8a7817a65dc7b1f6: Generated new TSK 0
I20260928 06:35:48.215811 29337 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.157.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab.
Entry for principal kudu/127.16.157.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab.
I20260928 06:35:48.894691 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.157.84:58421/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.65:0
--local_ip_for_outbound_sockets=127.16.157.65
--webserver_interface=127.16.157.65
--webserver_port=0
--tserver_master_addrs=127.16.157.126:44457
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab
--principal=kudu/127.16.157.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:35393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:49.013859 29368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:49.014174 29368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:49.014213 29368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:49.015894 29368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:49.015961 29368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:49.016001 29368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.65
I20260928 06:35:49.018029 29368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:35393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.16.157.84:58421/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab
--principal=kudu/127.16.157.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.157.65
--webserver_port=0
--tserver_master_addrs=127.16.157.126:44457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29368
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.65
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:35:49.018249 29368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:49.018504 29368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:49.027603 29374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:49.027617 29373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:49.027740 29376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:49.028215 29368 server_base.cc:1034] running on GCE node
I20260928 06:35:49.028456 29368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:49.028697 29368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:49.029871 29368 hybrid_clock.cc:648] HybridClock initialized: now 1790577349029849 us; error 41 us; skew 500 ppm
Sep 28 06:35:49 dist-test-slave-rrj5 krb5kdc[28811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577349, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:49.032553 29368 init.cc:377] Logged in from keytab as kudu/127.16.157.65@KRBTEST.COM (short username kudu)
I20260928 06:35:49.033020 29368 webserver.cc:460] Webserver started at http://127.16.157.65:42747/ using document root <none> and password file <none>
I20260928 06:35:49.033243 29368 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:49.033309 29368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:49.033464 29368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 06:35:49 dist-test-slave-rrj5 krb5kdc[28811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577349, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.65@KRBTEST.COM for HTTP/127.16.157.84@KRBTEST.COM
I20260928 06:35:49.046551 29368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-0/data/instance:
uuid: "74dd0463f0e346b8a2873f1911b61698"
format_stamp: "Formatted at 2026-09-28 06:35:49 on dist-test-slave-rrj5"
server_key: "e4a553788c6bf5866fc0985b6e43e4e7"
server_key_iv: "263c530350d441e32d2d692ad80ed533"
server_key_version: "kuduclusterkey@0"
I20260928 06:35:49.047108 29368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-0/wal/instance:
uuid: "74dd0463f0e346b8a2873f1911b61698"
format_stamp: "Formatted at 2026-09-28 06:35:49 on dist-test-slave-rrj5"
server_key: "e4a553788c6bf5866fc0985b6e43e4e7"
server_key_iv: "263c530350d441e32d2d692ad80ed533"
server_key_version: "kuduclusterkey@0"
I20260928 06:35:49.048985 29368 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260928 06:35:49.063488 29383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:49.063750 29368 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:35:49.063865 29368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-0/wal
uuid: "74dd0463f0e346b8a2873f1911b61698"
format_stamp: "Formatted at 2026-09-28 06:35:49 on dist-test-slave-rrj5"
server_key: "e4a553788c6bf5866fc0985b6e43e4e7"
server_key_iv: "263c530350d441e32d2d692ad80ed533"
server_key_version: "kuduclusterkey@0"
I20260928 06:35:49.064067 29368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:49.076571 29368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:49.079805 29368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:49.080037 29368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:49.080344 29368 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:49.080775 29368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:49.080868 29368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:49.080967 29368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:49.081041 29368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:49.088485 29368 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.65:41695
I20260928 06:35:49.088579 29498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.65:41695 every 8 connection(s)
I20260928 06:35:49.089033 29368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
I20260928 06:35:49.091300 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 29368
I20260928 06:35:49.091396 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-0/wal/instance
I20260928 06:35:49.091585 17013 external_mini_cluster.cc:1428] Setting key ce8f7952a641dfac45eab2714469cecd
Sep 28 06:35:49 dist-test-slave-rrj5 krb5kdc[28811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577349, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.65@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:35:49.104172 29501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:49.089819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:44457 (local address 127.16.157.65:49761)
0928 06:35:49.090285 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:49.090334 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:49.090973 (+ 639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:49.094394 (+ 3421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:49.094399 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:49.094674 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:49.094976 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:49.094983 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:49.095994 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:49.095995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:49.097302 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:49.097305 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:49.097910 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:49.097911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:49.098048 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:49.098661 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:49.098672 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:49.100371 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:49.102549 (+ 2178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:49.102555 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:49.102558 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:49.102871 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:49.103156 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:49.103157 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:49.103157 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:49.103286 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:49.103558 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:49.103564 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:49.103715 (+ 151us) client_negotiation.cc:769] Sending connection context
0928 06:35:49.103809 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 06:35:49.103956 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":86,"threads_started":1}
I20260928 06:35:49.104466 29502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:49.090115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:49761 (local address 127.16.157.126:44457)
0928 06:35:49.094128 (+ 4013us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:49.094130 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:49.094140 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:35:49.094165 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:49.094166 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:49.094179 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:49.094252 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:49.095092 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:49.095884 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:49.097406 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:49.097766 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:49.097831 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:49.100487 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:49.100488 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:49.100489 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:49.100496 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:49.102384 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:49.102977 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:49.102978 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:49.102979 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:49.103062 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:49.103362 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:49.103363 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:49.103363 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:49.103485 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:49.103529 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:49.104302 (+ 773us) server_negotiation.cc:299] Negotiation successful
0928 06:35:49.104386 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3979,"thread_start_us":81,"threads_started":1}
I20260928 06:35:49.105222 29499 heartbeater.cc:347] Connected to a master server at 127.16.157.126:44457
I20260928 06:35:49.105348 29499 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:49.105549 29499 heartbeater.cc:511] Master 127.16.157.126:44457 requested a full tablet report, sending...
I20260928 06:35:49.106215 29287 ts_manager.cc:194] Registered new tserver with Master: 74dd0463f0e346b8a2873f1911b61698 (127.16.157.65:41695)
I20260928 06:35:49.107075 29287 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.157.65@KRBTEST.COM'} at 127.16.157.65:49761
WARNING: no policy specified for kudu/127.16.157.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab.
Entry for principal kudu/127.16.157.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab.
I20260928 06:35:49.137140 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.157.84:58421/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.66:0
--local_ip_for_outbound_sockets=127.16.157.66
--webserver_interface=127.16.157.66
--webserver_port=0
--tserver_master_addrs=127.16.157.126:44457
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab
--principal=kudu/127.16.157.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:35393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:49.261242 29505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:49.261619 29505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:49.261726 29505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:49.263448 29505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:49.263581 29505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:49.263665 29505 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.66
I20260928 06:35:49.265915 29505 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:35393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.16.157.84:58421/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab
--principal=kudu/127.16.157.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.157.66
--webserver_port=0
--tserver_master_addrs=127.16.157.126:44457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.66
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:35:49.266392 29505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:49.266752 29505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:49.273715 29513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:49.274080 29512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:49.274246 29515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:49.278949 29505 server_base.cc:1034] running on GCE node
I20260928 06:35:49.279292 29505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:49.279592 29505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:49.280768 29505 hybrid_clock.cc:648] HybridClock initialized: now 1790577349280747 us; error 35 us; skew 500 ppm
Sep 28 06:35:49 dist-test-slave-rrj5 krb5kdc[28811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577349, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:49.284148 29505 init.cc:377] Logged in from keytab as kudu/127.16.157.66@KRBTEST.COM (short username kudu)
I20260928 06:35:49.284696 29505 webserver.cc:460] Webserver started at http://127.16.157.66:37383/ using document root <none> and password file <none>
I20260928 06:35:49.284976 29505 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:49.285046 29505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:49.285193 29505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 06:35:49 dist-test-slave-rrj5 krb5kdc[28811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577349, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.66@KRBTEST.COM for HTTP/127.16.157.84@KRBTEST.COM
I20260928 06:35:49.307300 29505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-1/data/instance:
uuid: "c85caf3fc6b841f083b08f47ff8c17fc"
format_stamp: "Formatted at 2026-09-28 06:35:49 on dist-test-slave-rrj5"
server_key: "0301475591641caa5bdb11f26a1cd62a"
server_key_iv: "ec13bfe4944710995093c900f2e3c887"
server_key_version: "kuduclusterkey@0"
I20260928 06:35:49.307749 29505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-1/wal/instance:
uuid: "c85caf3fc6b841f083b08f47ff8c17fc"
format_stamp: "Formatted at 2026-09-28 06:35:49 on dist-test-slave-rrj5"
server_key: "0301475591641caa5bdb11f26a1cd62a"
server_key_iv: "ec13bfe4944710995093c900f2e3c887"
server_key_version: "kuduclusterkey@0"
Sep 28 06:35:49 dist-test-slave-rrj5 krb5kdc[28811](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790577349, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:49.313073 29505 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.000s
I20260928 06:35:49.331858 29522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:49.332116 29505 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.000s sys 0.000s
I20260928 06:35:49.332211 29505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-1/wal
uuid: "c85caf3fc6b841f083b08f47ff8c17fc"
format_stamp: "Formatted at 2026-09-28 06:35:49 on dist-test-slave-rrj5"
server_key: "0301475591641caa5bdb11f26a1cd62a"
server_key_iv: "ec13bfe4944710995093c900f2e3c887"
server_key_version: "kuduclusterkey@0"
I20260928 06:35:49.332297 29505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:49.352636 29505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:49.355170 29505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:49.355396 29505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:49.355674 29505 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:49.356098 29505 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:49.356189 29505 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:49.356266 29505 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:49.356331 29505 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:49.375136 29505 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.66:38271
I20260928 06:35:49.375314 29635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.66:38271 every 8 connection(s)
I20260928 06:35:49.375562 29505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
I20260928 06:35:49.380015 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 29505
I20260928 06:35:49.380131 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-1/wal/instance
I20260928 06:35:49.380319 17013 external_mini_cluster.cc:1428] Setting key 292b6d7fbb4e368071f13bd84036fc00
Sep 28 06:35:49 dist-test-slave-rrj5 krb5kdc[28811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577349, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.66@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:35:49.387071 29638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:49.376286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:44457 (local address 127.16.157.66:59687)
0928 06:35:49.376564 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:49.376611 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:49.377215 (+ 604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:49.377524 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:49.377529 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:49.377809 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:49.378073 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:49.378079 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:49.379101 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:49.379102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:49.380351 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:49.380353 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:49.381045 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:49.381046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:49.381185 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:49.381787 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:49.381797 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:49.383365 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:49.385404 (+ 2039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:49.385411 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:49.385413 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:49.385736 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:49.385965 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:49.385966 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:49.385967 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:49.386098 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:49.386397 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:49.386403 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:49.386580 (+ 177us) client_negotiation.cc:769] Sending connection context
0928 06:35:49.386679 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 06:35:49.386841 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":188,"threads_started":1}
I20260928 06:35:49.387733 29502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:49.376306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.66:59687 (local address 127.16.157.126:44457)
0928 06:35:49.376363 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:49.376365 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:49.376710 (+ 345us) server_negotiation.cc:407] Connection header received
0928 06:35:49.377334 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:49.377335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:49.377352 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:49.377438 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:49.378179 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:49.379002 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:49.380470 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:49.380803 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:49.380842 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:49.383477 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:49.383478 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:49.383478 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:49.383486 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:49.385322 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:49.385809 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:49.385810 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:49.385818 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:49.385881 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:49.386169 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:49.386170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:49.386170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:49.386310 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:49.386369 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:49.387555 (+ 1186us) server_negotiation.cc:299] Negotiation successful
0928 06:35:49.387644 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 06:35:49.388118 29636 heartbeater.cc:347] Connected to a master server at 127.16.157.126:44457
I20260928 06:35:49.388267 29636 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:49.388525 29636 heartbeater.cc:511] Master 127.16.157.126:44457 requested a full tablet report, sending...
I20260928 06:35:49.388922 29287 ts_manager.cc:194] Registered new tserver with Master: c85caf3fc6b841f083b08f47ff8c17fc (127.16.157.66:38271)
I20260928 06:35:49.389413 29287 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.157.66@KRBTEST.COM'} at 127.16.157.66:59687
WARNING: no policy specified for kudu/127.16.157.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab.
Entry for principal kudu/127.16.157.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab.
I20260928 06:35:49.415357 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.157.84:58421/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.67:0
--local_ip_for_outbound_sockets=127.16.157.67
--webserver_interface=127.16.157.67
--webserver_port=0
--tserver_master_addrs=127.16.157.126:44457
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab
--principal=kudu/127.16.157.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:35393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:49.529858 29642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:49.530208 29642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:49.530313 29642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:49.532091 29642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:49.532200 29642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:49.532289 29642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.67
I20260928 06:35:49.534430 29642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:35393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.16.157.84:58421/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab
--principal=kudu/127.16.157.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.157.67
--webserver_port=0
--tserver_master_addrs=127.16.157.126:44457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.67
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:35:49.534840 29642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:49.535135 29642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:49.543030 29648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:49.543045 29647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:49.544283 29642 server_base.cc:1034] running on GCE node
W20260928 06:35:49.544508 29650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:49.544801 29642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:49.545138 29642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:49.546427 29642 hybrid_clock.cc:648] HybridClock initialized: now 1790577349546385 us; error 44 us; skew 500 ppm
Sep 28 06:35:49 dist-test-slave-rrj5 krb5kdc[28811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577349, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:49.550038 29642 init.cc:377] Logged in from keytab as kudu/127.16.157.67@KRBTEST.COM (short username kudu)
I20260928 06:35:49.550621 29642 webserver.cc:460] Webserver started at http://127.16.157.67:41897/ using document root <none> and password file <none>
I20260928 06:35:49.550948 29642 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:49.551021 29642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:49.551203 29642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 06:35:49 dist-test-slave-rrj5 krb5kdc[28811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577349, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.67@KRBTEST.COM for HTTP/127.16.157.84@KRBTEST.COM
I20260928 06:35:49.564621 29642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-2/data/instance:
uuid: "a513449bc7e74ab896d04e547cf5f5c4"
format_stamp: "Formatted at 2026-09-28 06:35:49 on dist-test-slave-rrj5"
server_key: "9737508c693b98238ed6ac71d4bab1d8"
server_key_iv: "0ac010f1c6eeb8b895d3e3307ade1e77"
server_key_version: "kuduclusterkey@0"
I20260928 06:35:49.565073 29642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-2/wal/instance:
uuid: "a513449bc7e74ab896d04e547cf5f5c4"
format_stamp: "Formatted at 2026-09-28 06:35:49 on dist-test-slave-rrj5"
server_key: "9737508c693b98238ed6ac71d4bab1d8"
server_key_iv: "0ac010f1c6eeb8b895d3e3307ade1e77"
server_key_version: "kuduclusterkey@0"
I20260928 06:35:49.570026 29642 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.003s
I20260928 06:35:49.584012 29657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:49.584321 29642 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:35:49.584414 29642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-2/wal
uuid: "a513449bc7e74ab896d04e547cf5f5c4"
format_stamp: "Formatted at 2026-09-28 06:35:49 on dist-test-slave-rrj5"
server_key: "9737508c693b98238ed6ac71d4bab1d8"
server_key_iv: "0ac010f1c6eeb8b895d3e3307ade1e77"
server_key_version: "kuduclusterkey@0"
I20260928 06:35:49.584502 29642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:49.616951 29642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:49.620072 29642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:49.620340 29642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:49.620819 29642 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:49.621240 29642 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:49.621340 29642 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:49.621418 29642 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:49.621476 29642 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:49.627748 29642 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.67:35367
I20260928 06:35:49.628213 29642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
I20260928 06:35:49.633392 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 29642
I20260928 06:35:49.633502 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-2/wal/instance
I20260928 06:35:49.633736 17013 external_mini_cluster.cc:1428] Setting key bd1d7aa64311b209a4fc865bfe909bf2
I20260928 06:35:49.639376 29770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.67:35367 every 8 connection(s)
Sep 28 06:35:49 dist-test-slave-rrj5 krb5kdc[28811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577349, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.67@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:35:49.655380 29502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:49.642479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:33421 (local address 127.16.157.126:44457)
0928 06:35:49.642528 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:49.642531 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:49.644791 (+ 2260us) server_negotiation.cc:407] Connection header received
0928 06:35:49.645508 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:49.645510 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:49.645521 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:49.645598 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:49.646571 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:49.647392 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:49.648934 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:49.649232 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:49.649270 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:49.652042 (+ 2772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:49.652044 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:49.652044 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:49.652052 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:49.653815 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:49.654293 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:49.654294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:49.654294 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:49.654364 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:49.654652 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:49.654653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:49.654654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:49.654784 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:49.654824 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:49.655194 (+ 370us) server_negotiation.cc:299] Negotiation successful
0928 06:35:49.655273 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 06:35:49.656165 29773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:49.642165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:44457 (local address 127.16.157.67:33421)
0928 06:35:49.644635 (+ 2470us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:49.644690 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:49.645360 (+ 670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:49.645813 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:49.645818 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:49.646117 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:49.646432 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:49.646439 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:49.647525 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:49.647526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:49.648835 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:49.648837 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:49.649383 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:49.649384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:49.649520 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:49.650174 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:49.650185 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:49.651933 (+ 1748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:49.653910 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:49.653916 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:49.653918 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:49.654202 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:49.654437 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:49.654438 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:49.654438 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:49.654580 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:49.654869 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:49.654875 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:49.655044 (+ 169us) client_negotiation.cc:769] Sending connection context
0928 06:35:49.655766 (+ 722us) client_negotiation.cc:240] Negotiation successful
0928 06:35:49.655929 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2280,"thread_start_us":101,"threads_started":1}
I20260928 06:35:49.656847 29771 heartbeater.cc:347] Connected to a master server at 127.16.157.126:44457
I20260928 06:35:49.656976 29771 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:49.657160 29771 heartbeater.cc:511] Master 127.16.157.126:44457 requested a full tablet report, sending...
I20260928 06:35:49.657547 29287 ts_manager.cc:194] Registered new tserver with Master: a513449bc7e74ab896d04e547cf5f5c4 (127.16.157.67:35367)
I20260928 06:35:49.658073 29287 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.157.67@KRBTEST.COM'} at 127.16.157.67:33421
I20260928 06:35:49.666958 17013 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:35:49.667793 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 29368
I20260928 06:35:49.675161 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 29505
I20260928 06:35:49.682471 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 29642
I20260928 06:35:49.687768 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 29255
I20260928 06:35:49.692539 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:44457
--webserver_interface=127.16.157.126
--webserver_port=37573
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=kudu/127.16.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:35393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.157.126:44457
--ranger_config_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.157.84:58421/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.157.84:58421/kms with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:49.773626 29774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:49.773866 29774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:49.773905 29774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:49.775131 29774 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:49.775198 29774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:49.775224 29774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:49.775254 29774 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:49.775285 29774 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:35:49.776919 29774 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:35393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.157.84:58421/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.16.157.126:44457
--ranger_config_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=kudu/127.16.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:44457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.157.126
--webserver_port=37573
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:35:49.777182 29774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:49.777392 29774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:49.782223 29782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:49.782224 29779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:49.782297 29780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:49.782567 29774 server_base.cc:1034] running on GCE node
I20260928 06:35:49.782882 29774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:49.783161 29774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:49.784332 29774 hybrid_clock.cc:648] HybridClock initialized: now 1790577349784301 us; error 48 us; skew 500 ppm
Sep 28 06:35:49 dist-test-slave-rrj5 krb5kdc[28811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577349, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:49.786887 29774 init.cc:377] Logged in from keytab as kudu/127.16.157.126@KRBTEST.COM (short username kudu)
I20260928 06:35:49.787285 29774 webserver.cc:460] Webserver started at http://127.16.157.126:37573/ using document root <none> and password file <none>
I20260928 06:35:49.787531 29774 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:49.787604 29774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 06:35:49 dist-test-slave-rrj5 krb5kdc[28811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577349, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.126@KRBTEST.COM for HTTP/127.16.157.84@KRBTEST.COM
I20260928 06:35:49.829943 29774 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:35:49.830853 29790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:49.831054 29774 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:35:49.831146 29774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/master-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/master-0/wal
uuid: "b26988d78de3439d8a7817a65dc7b1f6"
format_stamp: "Formatted at 2026-09-28 06:35:48 on dist-test-slave-rrj5"
server_key: "42685550e9074b57de75ec2db83e6ae8"
server_key_iv: "c21a82ac816daad519a21462debc4dce"
server_key_version: "kuduclusterkey@0"
I20260928 06:35:49.831594 29774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:49.852658 29774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:49.854739 29774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:49.854884 29774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:49.859601 29774 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.126:44457
I20260928 06:35:49.859680 29842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.126:44457 every 8 connection(s)
I20260928 06:35:49.860037 29774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
I20260928 06:35:49.861646 29843 sys_catalog.cc:263] Verifying existing consensus state
I20260928 06:35:49.862468 29843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b26988d78de3439d8a7817a65dc7b1f6: Bootstrap starting.
I20260928 06:35:49.868093 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 29774
I20260928 06:35:49.868374 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.157.84:58421/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.65:41695
--local_ip_for_outbound_sockets=127.16.157.65
--tserver_master_addrs=127.16.157.126:44457
--webserver_port=42747
--webserver_interface=127.16.157.65
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab
--principal=kudu/127.16.157.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:35393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.157.84:58421/kms with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:35:49.872392 29843 log.cc:824] T 00000000000000000000000000000000 P b26988d78de3439d8a7817a65dc7b1f6: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:49.874718 29843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b26988d78de3439d8a7817a65dc7b1f6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:35:49.875038 29843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b26988d78de3439d8a7817a65dc7b1f6: Bootstrap complete.
I20260928 06:35:49.878449 29843 raft_consensus.cc:348] T 00000000000000000000000000000000 P b26988d78de3439d8a7817a65dc7b1f6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b26988d78de3439d8a7817a65dc7b1f6" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 44457 } }
I20260928 06:35:49.878768 29843 raft_consensus.cc:729] T 00000000000000000000000000000000 P b26988d78de3439d8a7817a65dc7b1f6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b26988d78de3439d8a7817a65dc7b1f6, State: Initialized, Role: FOLLOWER
I20260928 06:35:49.878968 29843 consensus_queue.cc:260] T 00000000000000000000000000000000 P b26988d78de3439d8a7817a65dc7b1f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b26988d78de3439d8a7817a65dc7b1f6" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 44457 } }
I20260928 06:35:49.879045 29843 raft_consensus.cc:388] T 00000000000000000000000000000000 P b26988d78de3439d8a7817a65dc7b1f6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:35:49.879119 29843 raft_consensus.cc:482] T 00000000000000000000000000000000 P b26988d78de3439d8a7817a65dc7b1f6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:35:49.879168 29843 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b26988d78de3439d8a7817a65dc7b1f6 [term 1 FOLLOWER]: Advancing to term 2
I20260928 06:35:49.880446 29843 raft_consensus.cc:504] T 00000000000000000000000000000000 P b26988d78de3439d8a7817a65dc7b1f6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b26988d78de3439d8a7817a65dc7b1f6" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 44457 } }
I20260928 06:35:49.880549 29843 leader_election.cc:302] T 00000000000000000000000000000000 P b26988d78de3439d8a7817a65dc7b1f6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b26988d78de3439d8a7817a65dc7b1f6; no voters:
I20260928 06:35:49.880712 29843 leader_election.cc:288] T 00000000000000000000000000000000 P b26988d78de3439d8a7817a65dc7b1f6 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 06:35:49.880769 29847 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b26988d78de3439d8a7817a65dc7b1f6 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 06:35:49.880970 29847 raft_consensus.cc:686] T 00000000000000000000000000000000 P b26988d78de3439d8a7817a65dc7b1f6 [term 2 LEADER]: Becoming Leader. State: Replica: b26988d78de3439d8a7817a65dc7b1f6, State: Running, Role: LEADER
I20260928 06:35:49.881085 29847 consensus_queue.cc:237] T 00000000000000000000000000000000 P b26988d78de3439d8a7817a65dc7b1f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b26988d78de3439d8a7817a65dc7b1f6" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 44457 } }
I20260928 06:35:49.881103 29843 sys_catalog.cc:564] T 00000000000000000000000000000000 P b26988d78de3439d8a7817a65dc7b1f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:35:49.881347 29847 sys_catalog.cc:455] T 00000000000000000000000000000000 P b26988d78de3439d8a7817a65dc7b1f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b26988d78de3439d8a7817a65dc7b1f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b26988d78de3439d8a7817a65dc7b1f6" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 44457 } } }
I20260928 06:35:49.881400 29847 sys_catalog.cc:458] T 00000000000000000000000000000000 P b26988d78de3439d8a7817a65dc7b1f6 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:49.881495 29847 sys_catalog.cc:455] T 00000000000000000000000000000000 P b26988d78de3439d8a7817a65dc7b1f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b26988d78de3439d8a7817a65dc7b1f6. Latest consensus state: current_term: 2 leader_uuid: "b26988d78de3439d8a7817a65dc7b1f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b26988d78de3439d8a7817a65dc7b1f6" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 44457 } } }
I20260928 06:35:49.881559 29847 sys_catalog.cc:458] T 00000000000000000000000000000000 P b26988d78de3439d8a7817a65dc7b1f6 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:49.882169 29851 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:35:49.882643 29851 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:35:49.883025 29843 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 06:35:49.883474 29851 catalog_manager.cc:1259] Loaded cluster ID: 2b9a40c0a075474c8e1803f6ebad16ff
I20260928 06:35:49.883517 29851 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:35:49.884707 29851 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:35:49.886722 29851 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b26988d78de3439d8a7817a65dc7b1f6: Loaded TSK: 0
I20260928 06:35:49.887387 29851 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 06:35:49.980005 29845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:49.980260 29845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:49.980297 29845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:49.981920 29845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:49.981992 29845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:49.982031 29845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.65
I20260928 06:35:49.984074 29845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:35393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.157.84:58421/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab
--principal=kudu/127.16.157.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.65:41695
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.157.65
--webserver_port=42747
--tserver_master_addrs=127.16.157.126:44457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.65
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:35:49.984294 29845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:49.984572 29845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:49.990953 29883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:49.991253 29881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:49.991492 29880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:49.991461 29845 server_base.cc:1034] running on GCE node
I20260928 06:35:49.991750 29845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:49.992024 29845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:49.993197 29845 hybrid_clock.cc:648] HybridClock initialized: now 1790577349993166 us; error 44 us; skew 500 ppm
Sep 28 06:35:49 dist-test-slave-rrj5 krb5kdc[28811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577349, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:49.996057 29845 init.cc:377] Logged in from keytab as kudu/127.16.157.65@KRBTEST.COM (short username kudu)
I20260928 06:35:49.996588 29845 webserver.cc:460] Webserver started at http://127.16.157.65:42747/ using document root <none> and password file <none>
I20260928 06:35:49.996855 29845 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:49.996940 29845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 06:35:49 dist-test-slave-rrj5 krb5kdc[28811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577349, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.65@KRBTEST.COM for HTTP/127.16.157.84@KRBTEST.COM
I20260928 06:35:50.018244 29845 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:35:50.019222 29891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:50.019382 29845 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:35:50.019469 29845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-0/wal
uuid: "74dd0463f0e346b8a2873f1911b61698"
format_stamp: "Formatted at 2026-09-28 06:35:49 on dist-test-slave-rrj5"
server_key: "e4a553788c6bf5866fc0985b6e43e4e7"
server_key_iv: "263c530350d441e32d2d692ad80ed533"
server_key_version: "kuduclusterkey@0"
I20260928 06:35:50.019934 29845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:50.042330 29845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:50.045890 29845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:50.046046 29845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:50.046267 29845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:50.046731 29845 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:50.046814 29845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:50.046849 29845 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:50.046903 29845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:50.054023 29845 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.65:41695
I20260928 06:35:50.054255 30004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.65:41695 every 8 connection(s)
I20260928 06:35:50.054503 29845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
I20260928 06:35:50.056429 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 29845
I20260928 06:35:50.056718 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.157.84:58421/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.66:38271
--local_ip_for_outbound_sockets=127.16.157.66
--tserver_master_addrs=127.16.157.126:44457
--webserver_port=37383
--webserver_interface=127.16.157.66
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab
--principal=kudu/127.16.157.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:35393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.157.84:58421/kms with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 06:35:50 dist-test-slave-rrj5 krb5kdc[28811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577349, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.65@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:35:50.083315 30008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:50.055476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:60193 (local address 127.16.157.126:44457)
0928 06:35:50.062282 (+ 6806us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:50.062292 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:50.062349 (+ 57us) server_negotiation.cc:407] Connection header received
0928 06:35:50.062518 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:50.062526 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:50.062590 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:50.062743 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:50.063599 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:50.065058 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:50.075120 (+ 10062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:50.075775 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:50.075886 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:50.078687 (+ 2801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:50.078692 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:50.078692 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:50.078703 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:50.080744 (+ 2041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:50.081232 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:50.081236 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:50.081236 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:50.081353 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:50.081653 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:50.081654 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:50.081654 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:50.082023 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:50.082104 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:50.082986 (+ 882us) server_negotiation.cc:299] Negotiation successful
0928 06:35:50.083111 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6743,"thread_start_us":81,"threads_started":1}
I20260928 06:35:50.082798 30007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:50.055213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:44457 (local address 127.16.157.65:60193)
0928 06:35:50.055385 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:50.055435 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:50.056090 (+ 655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:50.062918 (+ 6828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:50.062922 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:50.063158 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:50.063461 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:50.063468 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:50.073655 (+ 10187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:50.073657 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:50.074990 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:50.074995 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:50.076012 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:50.076013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:50.076149 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:50.076784 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:50.076795 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:50.078558 (+ 1763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:50.080840 (+ 2282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:50.080846 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:50.080848 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:50.081137 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:50.081430 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:50.081431 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:50.081431 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:50.081557 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:50.082133 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:50.082139 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:50.082285 (+ 146us) client_negotiation.cc:769] Sending connection context
0928 06:35:50.082406 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 06:35:50.082559 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":89,"threads_started":1}
I20260928 06:35:50.084616 30005 heartbeater.cc:347] Connected to a master server at 127.16.157.126:44457
I20260928 06:35:50.084715 30005 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:50.084890 30005 heartbeater.cc:511] Master 127.16.157.126:44457 requested a full tablet report, sending...
W20260928 06:35:50.085816 30005 heartbeater.cc:644] Failed to heartbeat to 127.16.157.126:44457 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 06:35:50.085896 30005 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:50.085966 30005 heartbeater.cc:511] Master 127.16.157.126:44457 requested a full tablet report, sending...
I20260928 06:35:50.086287 30005 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:50.086373 30005 heartbeater.cc:511] Master 127.16.157.126:44457 requested a full tablet report, sending...
W20260928 06:35:50.086750 30005 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 06:35:50.164405 30009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:50.164685 30009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:50.164719 30009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:50.166509 30009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:50.166571 30009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:50.166611 30009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.66
I20260928 06:35:50.168272 30009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:35393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.157.84:58421/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab
--principal=kudu/127.16.157.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.66:38271
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.157.66
--webserver_port=37383
--tserver_master_addrs=127.16.157.126:44457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.66
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:35:50.168540 30009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:50.169054 30009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:50.174590 30015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:50.174702 30014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:50.174578 30017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:50.175249 30009 server_base.cc:1034] running on GCE node
I20260928 06:35:50.175483 30009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:50.175757 30009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:50.179031 30009 hybrid_clock.cc:648] HybridClock initialized: now 1790577350179000 us; error 44 us; skew 500 ppm
Sep 28 06:35:50 dist-test-slave-rrj5 krb5kdc[28811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577350, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:50.182543 30009 init.cc:377] Logged in from keytab as kudu/127.16.157.66@KRBTEST.COM (short username kudu)
I20260928 06:35:50.183100 30009 webserver.cc:460] Webserver started at http://127.16.157.66:37383/ using document root <none> and password file <none>
I20260928 06:35:50.183441 30009 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:50.183527 30009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 06:35:50 dist-test-slave-rrj5 krb5kdc[28811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577350, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.66@KRBTEST.COM for HTTP/127.16.157.84@KRBTEST.COM
I20260928 06:35:50.217484 30009 fs_manager.cc:709] Time spent opening directory manager: real 0.017s user 0.001s sys 0.000s
I20260928 06:35:50.218887 30025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:50.219055 30009 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:35:50.219138 30009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-1/wal
uuid: "c85caf3fc6b841f083b08f47ff8c17fc"
format_stamp: "Formatted at 2026-09-28 06:35:49 on dist-test-slave-rrj5"
server_key: "0301475591641caa5bdb11f26a1cd62a"
server_key_iv: "ec13bfe4944710995093c900f2e3c887"
server_key_version: "kuduclusterkey@0"
I20260928 06:35:50.219599 30009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:50.233832 30009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:50.236481 30009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:50.236652 30009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:50.236943 30009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:50.237345 30009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:50.237394 30009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:50.237426 30009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:50.237488 30009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:50.245165 30009 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.66:38271
I20260928 06:35:50.245285 30138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.66:38271 every 8 connection(s)
I20260928 06:35:50.245795 30009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
Sep 28 06:35:50 dist-test-slave-rrj5 krb5kdc[28811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577350, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.66@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:35:50.255223 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 30009
I20260928 06:35:50.255549 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.157.84:58421/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.67:35367
--local_ip_for_outbound_sockets=127.16.157.67
--tserver_master_addrs=127.16.157.126:44457
--webserver_port=41897
--webserver_interface=127.16.157.67
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab
--principal=kudu/127.16.157.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:35393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.157.84:58421/kms with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:35:50.256908 30008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:50.246681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.66:54865 (local address 127.16.157.126:44457)
0928 06:35:50.246730 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:50.246733 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:50.247074 (+ 341us) server_negotiation.cc:407] Connection header received
0928 06:35:50.247719 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:50.247720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:50.247735 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:50.247827 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:50.248372 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:50.249173 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:50.250322 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:50.250634 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:50.250721 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:50.253083 (+ 2362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:50.253085 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:50.253085 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:50.253093 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:50.254984 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:50.255510 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:50.255512 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:50.255512 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:50.255581 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:50.255970 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:50.255971 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:50.255971 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:50.256138 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:50.256213 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:50.256608 (+ 395us) server_negotiation.cc:299] Negotiation successful
0928 06:35:50.256760 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:35:50.257903 30141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:50.246700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:44457 (local address 127.16.157.66:54865)
0928 06:35:50.246906 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:50.246958 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:50.247597 (+ 639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:50.247884 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:50.247887 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:50.248065 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:50.248270 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:50.248274 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:50.249277 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:50.249278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:50.250229 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:50.250231 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:50.250752 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:50.250754 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:50.250857 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:50.251466 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:50.251474 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:50.252961 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:50.255081 (+ 2120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:50.255090 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:50.255093 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:50.255422 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:50.255678 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:50.255679 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:50.255679 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:50.255838 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:50.256244 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:50.256252 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:50.256437 (+ 185us) client_negotiation.cc:769] Sending connection context
0928 06:35:50.257358 (+ 921us) client_negotiation.cc:240] Negotiation successful
0928 06:35:50.257560 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":106,"threads_started":1}
I20260928 06:35:50.258728 30139 heartbeater.cc:347] Connected to a master server at 127.16.157.126:44457
I20260928 06:35:50.258841 30139 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:50.259087 30139 heartbeater.cc:511] Master 127.16.157.126:44457 requested a full tablet report, sending...
W20260928 06:35:50.259981 30139 heartbeater.cc:644] Failed to heartbeat to 127.16.157.126:44457 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 06:35:50.260063 30139 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:50.260139 30139 heartbeater.cc:511] Master 127.16.157.126:44457 requested a full tablet report, sending...
I20260928 06:35:50.262622 30139 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:50.262732 30139 heartbeater.cc:511] Master 127.16.157.126:44457 requested a full tablet report, sending...
W20260928 06:35:50.263170 30139 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 06:35:50.362063 30142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:50.362355 30142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:50.362392 30142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:50.364122 30142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:50.364255 30142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:50.364351 30142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.67
I20260928 06:35:50.366640 30142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:35393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.157.84:58421/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab
--principal=kudu/127.16.157.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.67:35367
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.157.67
--webserver_port=41897
--tserver_master_addrs=127.16.157.126:44457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.67
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:35:50.367148 30142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:50.367472 30142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:50.374758 30148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:50.374864 30142 server_base.cc:1034] running on GCE node
W20260928 06:35:50.374629 30149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:50.377779 30151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:50.378063 30142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:50.378374 30142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:50.379570 30142 hybrid_clock.cc:648] HybridClock initialized: now 1790577350379532 us; error 52 us; skew 500 ppm
Sep 28 06:35:50 dist-test-slave-rrj5 krb5kdc[28811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577350, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:50.383692 30142 init.cc:377] Logged in from keytab as kudu/127.16.157.67@KRBTEST.COM (short username kudu)
I20260928 06:35:50.384346 30142 webserver.cc:460] Webserver started at http://127.16.157.67:41897/ using document root <none> and password file <none>
I20260928 06:35:50.384630 30142 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:50.384812 30142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 06:35:50 dist-test-slave-rrj5 krb5kdc[28811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577350, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.67@KRBTEST.COM for HTTP/127.16.157.84@KRBTEST.COM
I20260928 06:35:50.407847 30142 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:35:50.408859 30159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:50.409075 30142 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:35:50.409171 30142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-2/wal
uuid: "a513449bc7e74ab896d04e547cf5f5c4"
format_stamp: "Formatted at 2026-09-28 06:35:49 on dist-test-slave-rrj5"
server_key: "9737508c693b98238ed6ac71d4bab1d8"
server_key_iv: "0ac010f1c6eeb8b895d3e3307ade1e77"
server_key_version: "kuduclusterkey@0"
I20260928 06:35:50.409626 30142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:50.418644 30142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:50.420970 30142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:50.421141 30142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:50.421382 30142 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:50.421784 30142 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:50.421840 30142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:50.421885 30142 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:50.421912 30142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:50.429664 30142 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.67:35367
I20260928 06:35:50.430240 30142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
I20260928 06:35:50.432775 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 30142
Sep 28 06:35:50 dist-test-slave-rrj5 krb5kdc[28811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577350, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.67@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:35:50.440843 30272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.67:35367 every 8 connection(s)
I20260928 06:35:50.443161 30275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:50.431102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:44457 (local address 127.16.157.67:33243)
0928 06:35:50.431283 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:50.431333 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:50.431991 (+ 658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:50.432312 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:50.432316 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:50.432560 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:50.432961 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:50.432969 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:50.434104 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:50.434105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:50.435499 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:50.435502 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:50.436083 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:50.436085 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:50.436237 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:50.437540 (+ 1303us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:50.437596 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:50.439340 (+ 1744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:50.441275 (+ 1935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:50.441282 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:50.441285 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:50.441637 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:50.441864 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:50.441865 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:50.441865 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:50.442029 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:50.442342 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:50.442350 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:50.442548 (+ 198us) client_negotiation.cc:769] Sending connection context
0928 06:35:50.442689 (+ 141us) client_negotiation.cc:240] Negotiation successful
0928 06:35:50.442883 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":100,"threads_started":1}
I20260928 06:35:50.443215 30008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:50.431116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:33243 (local address 127.16.157.126:44457)
0928 06:35:50.431169 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:50.431171 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:50.431453 (+ 282us) server_negotiation.cc:407] Connection header received
0928 06:35:50.432138 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:50.432140 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:50.432154 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:50.432229 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:50.433084 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:50.433935 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:50.435593 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:50.435910 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:50.435991 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:50.439459 (+ 3468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:50.439460 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:50.439461 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:50.439468 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:50.441172 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:50.441714 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:50.441716 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:50.441716 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:50.441782 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:50.442114 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:50.442115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:50.442115 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:50.442263 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:50.442331 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:50.442986 (+ 655us) server_negotiation.cc:299] Negotiation successful
0928 06:35:50.443121 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 06:35:50.443920 30273 heartbeater.cc:347] Connected to a master server at 127.16.157.126:44457
I20260928 06:35:50.444031 30273 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:50.444228 30273 heartbeater.cc:511] Master 127.16.157.126:44457 requested a full tablet report, sending...
W20260928 06:35:50.444878 30273 heartbeater.cc:644] Failed to heartbeat to 127.16.157.126:44457 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 06:35:50.444970 30273 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:50.445040 30273 heartbeater.cc:511] Master 127.16.157.126:44457 requested a full tablet report, sending...
I20260928 06:35:50.445387 30273 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:50.445465 30273 heartbeater.cc:511] Master 127.16.157.126:44457 requested a full tablet report, sending...
W20260928 06:35:50.445773 30273 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 06:35:50.449050 30277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:50.434099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58010 (local address 127.16.157.126:44457)
0928 06:35:50.435835 (+ 1736us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:50.435837 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:50.435844 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:35:50.435865 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:50.435865 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:50.435878 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:50.435935 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:50.436800 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:50.437633 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:50.438771 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:50.439075 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:50.439150 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:50.441843 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:50.441845 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:50.441845 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:50.441853 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:50.443843 (+ 1990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:50.448170 (+ 4327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:50.448171 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:50.448171 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:50.448237 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:50.448496 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:50.448497 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:50.448497 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:50.448603 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:50.448728 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:50.448831 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 06:35:50.448950 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1717,"thread_start_us":72,"threads_started":1}
Sep 28 06:35:50 dist-test-slave-rrj5 krb5kdc[28811](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790577350, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:51.086915 30005 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:51.087085 30005 heartbeater.cc:511] Master 127.16.157.126:44457 requested a full tablet report, sending...
I20260928 06:35:51.087491 29807 ts_manager.cc:194] Registered new tserver with Master: 74dd0463f0e346b8a2873f1911b61698 (127.16.157.65:41695)
I20260928 06:35:51.088536 29807 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.157.65@KRBTEST.COM'} at 127.16.157.65:60193
I20260928 06:35:51.263382 30139 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:51.263545 30139 heartbeater.cc:511] Master 127.16.157.126:44457 requested a full tablet report, sending...
I20260928 06:35:51.263911 29807 ts_manager.cc:194] Registered new tserver with Master: c85caf3fc6b841f083b08f47ff8c17fc (127.16.157.66:38271)
I20260928 06:35:51.264474 29807 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.157.66@KRBTEST.COM'} at 127.16.157.66:54865
Sep 28 06:35:51 dist-test-slave-rrj5 krb5kdc[28811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577350, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.126@KRBTEST.COM for HTTP/127.16.157.84@KRBTEST.COM
I20260928 06:35:51.445945 30273 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:51.446094 30273 heartbeater.cc:511] Master 127.16.157.126:44457 requested a full tablet report, sending...
I20260928 06:35:51.446376 29807 ts_manager.cc:194] Registered new tserver with Master: a513449bc7e74ab896d04e547cf5f5c4 (127.16.157.67:35367)
I20260928 06:35:51.446894 29807 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.157.67@KRBTEST.COM'} at 127.16.157.67:33243
I20260928 06:35:51.455984 17013 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:35:51.465549 30295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:51.456978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58022 (local address 127.16.157.126:44457)
0928 06:35:51.457871 (+ 893us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:51.457873 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:51.457880 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:35:51.457902 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:51.457903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:51.457915 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:51.457986 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:51.458702 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:51.459505 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:51.460562 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:51.460894 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:51.460963 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:51.462469 (+ 1506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:51.462470 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:51.462470 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:51.462478 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:51.464295 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:51.464687 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:51.464688 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:51.464688 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:51.464757 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:51.464977 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:51.464978 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:51.464978 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:51.465118 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:51.465239 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:51.465358 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 06:35:51.465445 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":871,"thread_start_us":71,"threads_started":1}
I20260928 06:35:51.468299 29807 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58022:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:35:51.471328 29807 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:35:51.484128 30301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:51.474706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.65:41695 (local address 127.0.0.1:38598)
0928 06:35:51.479314 (+ 4608us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:51.479325 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:51.479368 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:51.479811 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:51.479812 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:51.479814 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:51.479859 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:51.479860 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:51.481357 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:51.481357 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:51.482523 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:51.482525 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:51.483859 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:51.483860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:51.483934 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:51.483949 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 06:35:51.483997 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 06:35:51.484044 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4587,"mutex_wait_us":2,"thread_start_us":71,"threads_started":1}
I20260928 06:35:51.484544 30300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:51.476139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.67:35367 (local address 127.0.0.1:42106)
0928 06:35:51.479452 (+ 3313us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:51.479460 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:51.479498 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:51.480938 (+ 1440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:51.480939 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:51.480940 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:51.480975 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:51.480976 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:51.482248 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:51.482249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:51.483283 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:51.483286 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:51.484308 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:51.484309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:51.484378 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:51.484382 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:35:51.484427 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 06:35:51.484463 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3301,"thread_start_us":92,"threads_started":1}
I20260928 06:35:51.485065 30302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:51.474974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38598 (local address 127.16.157.65:41695)
0928 06:35:51.479597 (+ 4623us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:51.479601 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:51.479610 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:35:51.479642 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:51.479644 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:51.479659 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:51.479727 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:51.479937 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:51.481266 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:51.482622 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:51.483739 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:51.484858 (+ 1119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:51.484924 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:51.484951 (+ 27us) server_negotiation.cc:299] Negotiation successful
0928 06:35:51.484990 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4588,"thread_start_us":52,"threads_started":1}
I20260928 06:35:51.485903 30303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:51.475298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33228 (local address 127.16.157.66:38271)
0928 06:35:51.476695 (+ 1397us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:51.476700 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:51.476953 (+ 253us) server_negotiation.cc:407] Connection header received
0928 06:35:51.477152 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:51.477154 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:51.477172 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:51.477297 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:51.477755 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:51.479165 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:51.484663 (+ 5498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:51.485454 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:51.485502 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:51.485553 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:51.485781 (+ 228us) server_negotiation.cc:299] Negotiation successful
0928 06:35:51.485824 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1362,"thread_start_us":630,"threads_started":1}
I20260928 06:35:51.485895 29931 tablet_service.cc:1467] Processing CreateTablet for tablet 1e40c426f2a847c0a98bf0b666896bd6 (DEFAULT_TABLE table=test-table [id=a8e523b2d3eb45538aa2894e0a1ee1f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:51.486084 30305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:51.474706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.66:38271 (local address 127.0.0.1:33228)
0928 06:35:51.476791 (+ 2085us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:51.476867 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:51.476996 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:51.477445 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:51.477488 (+ 43us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:51.477491 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:51.477596 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:51.477599 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:51.479260 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:51.479261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:51.480590 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:51.480593 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:51.485632 (+ 5039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:51.485633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:51.485691 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:51.485695 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:35:51.485978 (+ 283us) client_negotiation.cc:240] Negotiation successful
0928 06:35:51.486022 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2054,"thread_start_us":78,"threads_started":1}
I20260928 06:35:51.486254 29931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e40c426f2a847c0a98bf0b666896bd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:51.486300 30304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:51.475631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42106 (local address 127.16.157.67:35367)
0928 06:35:51.478609 (+ 2978us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:51.478614 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:51.480679 (+ 2065us) server_negotiation.cc:407] Connection header received
0928 06:35:51.480753 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:51.480756 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:51.480773 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:51.480848 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:51.481042 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:51.482156 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:51.483398 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:51.484146 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:51.484211 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:51.484277 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:51.486174 (+ 1897us) server_negotiation.cc:299] Negotiation successful
0928 06:35:51.486220 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2945,"thread_start_us":103,"threads_started":1}
I20260928 06:35:51.488929 30306 tablet_bootstrap.cc:492] T 1e40c426f2a847c0a98bf0b666896bd6 P 74dd0463f0e346b8a2873f1911b61698: Bootstrap starting.
I20260928 06:35:51.489638 30306 tablet_bootstrap.cc:654] T 1e40c426f2a847c0a98bf0b666896bd6 P 74dd0463f0e346b8a2873f1911b61698: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:51.489936 30306 log.cc:824] T 1e40c426f2a847c0a98bf0b666896bd6 P 74dd0463f0e346b8a2873f1911b61698: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:51.489894 30190 tablet_service.cc:1467] Processing CreateTablet for tablet 1e40c426f2a847c0a98bf0b666896bd6 (DEFAULT_TABLE table=test-table [id=a8e523b2d3eb45538aa2894e0a1ee1f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:51.490216 30190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e40c426f2a847c0a98bf0b666896bd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:51.492452 30308 tablet_bootstrap.cc:492] T 1e40c426f2a847c0a98bf0b666896bd6 P a513449bc7e74ab896d04e547cf5f5c4: Bootstrap starting.
I20260928 06:35:51.493039 30308 tablet_bootstrap.cc:654] T 1e40c426f2a847c0a98bf0b666896bd6 P a513449bc7e74ab896d04e547cf5f5c4: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:51.493100 30057 tablet_service.cc:1467] Processing CreateTablet for tablet 1e40c426f2a847c0a98bf0b666896bd6 (DEFAULT_TABLE table=test-table [id=a8e523b2d3eb45538aa2894e0a1ee1f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:51.493335 30308 log.cc:824] T 1e40c426f2a847c0a98bf0b666896bd6 P a513449bc7e74ab896d04e547cf5f5c4: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:51.493427 30057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e40c426f2a847c0a98bf0b666896bd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:51.494776 30306 tablet_bootstrap.cc:492] T 1e40c426f2a847c0a98bf0b666896bd6 P 74dd0463f0e346b8a2873f1911b61698: No bootstrap required, opened a new log
I20260928 06:35:51.494848 30306 ts_tablet_manager.cc:1397] T 1e40c426f2a847c0a98bf0b666896bd6 P 74dd0463f0e346b8a2873f1911b61698: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20260928 06:35:51.497044 30306 raft_consensus.cc:348] T 1e40c426f2a847c0a98bf0b666896bd6 P 74dd0463f0e346b8a2873f1911b61698 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74dd0463f0e346b8a2873f1911b61698" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 41695 } } peers { permanent_uuid: "a513449bc7e74ab896d04e547cf5f5c4" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35367 } } peers { permanent_uuid: "c85caf3fc6b841f083b08f47ff8c17fc" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 38271 } }
I20260928 06:35:51.497215 30306 raft_consensus.cc:374] T 1e40c426f2a847c0a98bf0b666896bd6 P 74dd0463f0e346b8a2873f1911b61698 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:51.497256 30306 raft_consensus.cc:729] T 1e40c426f2a847c0a98bf0b666896bd6 P 74dd0463f0e346b8a2873f1911b61698 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74dd0463f0e346b8a2873f1911b61698, State: Initialized, Role: FOLLOWER
I20260928 06:35:51.497404 30306 consensus_queue.cc:260] T 1e40c426f2a847c0a98bf0b666896bd6 P 74dd0463f0e346b8a2873f1911b61698 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74dd0463f0e346b8a2873f1911b61698" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 41695 } } peers { permanent_uuid: "a513449bc7e74ab896d04e547cf5f5c4" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35367 } } peers { permanent_uuid: "c85caf3fc6b841f083b08f47ff8c17fc" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 38271 } }
I20260928 06:35:51.497676 30306 ts_tablet_manager.cc:1428] T 1e40c426f2a847c0a98bf0b666896bd6 P 74dd0463f0e346b8a2873f1911b61698: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 06:35:51.497695 30005 heartbeater.cc:503] Master 127.16.157.126:44457 was elected leader, sending a full tablet report...
I20260928 06:35:51.498164 30308 tablet_bootstrap.cc:492] T 1e40c426f2a847c0a98bf0b666896bd6 P a513449bc7e74ab896d04e547cf5f5c4: No bootstrap required, opened a new log
I20260928 06:35:51.498226 30308 ts_tablet_manager.cc:1397] T 1e40c426f2a847c0a98bf0b666896bd6 P a513449bc7e74ab896d04e547cf5f5c4: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260928 06:35:51.500116 30308 raft_consensus.cc:348] T 1e40c426f2a847c0a98bf0b666896bd6 P a513449bc7e74ab896d04e547cf5f5c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74dd0463f0e346b8a2873f1911b61698" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 41695 } } peers { permanent_uuid: "a513449bc7e74ab896d04e547cf5f5c4" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35367 } } peers { permanent_uuid: "c85caf3fc6b841f083b08f47ff8c17fc" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 38271 } }
I20260928 06:35:51.500218 30308 raft_consensus.cc:374] T 1e40c426f2a847c0a98bf0b666896bd6 P a513449bc7e74ab896d04e547cf5f5c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:51.500237 30308 raft_consensus.cc:729] T 1e40c426f2a847c0a98bf0b666896bd6 P a513449bc7e74ab896d04e547cf5f5c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a513449bc7e74ab896d04e547cf5f5c4, State: Initialized, Role: FOLLOWER
I20260928 06:35:51.500356 30308 consensus_queue.cc:260] T 1e40c426f2a847c0a98bf0b666896bd6 P a513449bc7e74ab896d04e547cf5f5c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74dd0463f0e346b8a2873f1911b61698" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 41695 } } peers { permanent_uuid: "a513449bc7e74ab896d04e547cf5f5c4" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35367 } } peers { permanent_uuid: "c85caf3fc6b841f083b08f47ff8c17fc" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 38271 } }
I20260928 06:35:51.500604 30308 ts_tablet_manager.cc:1428] T 1e40c426f2a847c0a98bf0b666896bd6 P a513449bc7e74ab896d04e547cf5f5c4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:35:51.500615 30273 heartbeater.cc:503] Master 127.16.157.126:44457 was elected leader, sending a full tablet report...
I20260928 06:35:51.501691 30310 tablet_bootstrap.cc:492] T 1e40c426f2a847c0a98bf0b666896bd6 P c85caf3fc6b841f083b08f47ff8c17fc: Bootstrap starting.
I20260928 06:35:51.502321 30310 tablet_bootstrap.cc:654] T 1e40c426f2a847c0a98bf0b666896bd6 P c85caf3fc6b841f083b08f47ff8c17fc: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:51.502636 30310 log.cc:824] T 1e40c426f2a847c0a98bf0b666896bd6 P c85caf3fc6b841f083b08f47ff8c17fc: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:51.503348 30310 tablet_bootstrap.cc:492] T 1e40c426f2a847c0a98bf0b666896bd6 P c85caf3fc6b841f083b08f47ff8c17fc: No bootstrap required, opened a new log
I20260928 06:35:51.503429 30310 ts_tablet_manager.cc:1397] T 1e40c426f2a847c0a98bf0b666896bd6 P c85caf3fc6b841f083b08f47ff8c17fc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:35:51.504910 30310 raft_consensus.cc:348] T 1e40c426f2a847c0a98bf0b666896bd6 P c85caf3fc6b841f083b08f47ff8c17fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74dd0463f0e346b8a2873f1911b61698" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 41695 } } peers { permanent_uuid: "a513449bc7e74ab896d04e547cf5f5c4" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35367 } } peers { permanent_uuid: "c85caf3fc6b841f083b08f47ff8c17fc" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 38271 } }
I20260928 06:35:51.505054 30310 raft_consensus.cc:374] T 1e40c426f2a847c0a98bf0b666896bd6 P c85caf3fc6b841f083b08f47ff8c17fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:51.505127 30310 raft_consensus.cc:729] T 1e40c426f2a847c0a98bf0b666896bd6 P c85caf3fc6b841f083b08f47ff8c17fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c85caf3fc6b841f083b08f47ff8c17fc, State: Initialized, Role: FOLLOWER
I20260928 06:35:51.505268 30310 consensus_queue.cc:260] T 1e40c426f2a847c0a98bf0b666896bd6 P c85caf3fc6b841f083b08f47ff8c17fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74dd0463f0e346b8a2873f1911b61698" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 41695 } } peers { permanent_uuid: "a513449bc7e74ab896d04e547cf5f5c4" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35367 } } peers { permanent_uuid: "c85caf3fc6b841f083b08f47ff8c17fc" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 38271 } }
I20260928 06:35:51.505548 30139 heartbeater.cc:503] Master 127.16.157.126:44457 was elected leader, sending a full tablet report...
I20260928 06:35:51.505543 30310 ts_tablet_manager.cc:1428] T 1e40c426f2a847c0a98bf0b666896bd6 P c85caf3fc6b841f083b08f47ff8c17fc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260928 06:35:51.556802 30006 tablet.cc:2369] T 1e40c426f2a847c0a98bf0b666896bd6 P 74dd0463f0e346b8a2873f1911b61698: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 06:35:51 dist-test-slave-rrj5 krb5kdc[28811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577350, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.126@KRBTEST.COM for HTTP/127.16.157.84@KRBTEST.COM
W20260928 06:35:51.681727 30274 tablet.cc:2369] T 1e40c426f2a847c0a98bf0b666896bd6 P a513449bc7e74ab896d04e547cf5f5c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:35:51.748096 30140 tablet.cc:2369] T 1e40c426f2a847c0a98bf0b666896bd6 P c85caf3fc6b841f083b08f47ff8c17fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:35:51.856568 30311 raft_consensus.cc:482] T 1e40c426f2a847c0a98bf0b666896bd6 P 74dd0463f0e346b8a2873f1911b61698 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:35:51.856725 30311 raft_consensus.cc:504] T 1e40c426f2a847c0a98bf0b666896bd6 P 74dd0463f0e346b8a2873f1911b61698 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74dd0463f0e346b8a2873f1911b61698" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 41695 } } peers { permanent_uuid: "a513449bc7e74ab896d04e547cf5f5c4" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35367 } } peers { permanent_uuid: "c85caf3fc6b841f083b08f47ff8c17fc" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 38271 } }
I20260928 06:35:51.857206 30311 leader_election.cc:288] T 1e40c426f2a847c0a98bf0b666896bd6 P 74dd0463f0e346b8a2873f1911b61698 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a513449bc7e74ab896d04e547cf5f5c4 (127.16.157.67:35367), c85caf3fc6b841f083b08f47ff8c17fc (127.16.157.66:38271)
I20260928 06:35:51.860996 30324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:51.857439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.66:38271 (local address 127.16.157.65:33271)
0928 06:35:51.857790 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:51.857801 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:51.857890 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:51.858083 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:51.858084 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:51.858086 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:51.858178 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:51.858180 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:51.859019 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:51.859020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:51.859892 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:51.859895 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:51.860694 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:51.860695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:51.860771 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:51.860774 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:35:51.860838 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 06:35:51.860890 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":76,"threads_started":1}
I20260928 06:35:51.861006 30303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:51.857672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:33271 (local address 127.16.157.66:38271)
0928 06:35:51.857726 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:51.857728 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:51.857862 (+ 134us) server_negotiation.cc:407] Connection header received
0928 06:35:51.857975 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:51.857977 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:51.857989 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:51.858066 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:51.858303 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:51.858913 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:51.859981 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:51.860568 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:51.860673 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:51.860684 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:51.860891 (+ 207us) server_negotiation.cc:299] Negotiation successful
0928 06:35:51.860945 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 06:35:51.861493 30080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e40c426f2a847c0a98bf0b666896bd6" candidate_uuid: "74dd0463f0e346b8a2873f1911b61698" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c85caf3fc6b841f083b08f47ff8c17fc" is_pre_election: true
I20260928 06:35:51.861671 30080 raft_consensus.cc:2454] T 1e40c426f2a847c0a98bf0b666896bd6 P c85caf3fc6b841f083b08f47ff8c17fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74dd0463f0e346b8a2873f1911b61698 in term 0.
I20260928 06:35:51.861922 29894 leader_election.cc:302] T 1e40c426f2a847c0a98bf0b666896bd6 P 74dd0463f0e346b8a2873f1911b61698 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74dd0463f0e346b8a2873f1911b61698, c85caf3fc6b841f083b08f47ff8c17fc; no voters:
I20260928 06:35:51.862296 30311 raft_consensus.cc:2781] T 1e40c426f2a847c0a98bf0b666896bd6 P 74dd0463f0e346b8a2873f1911b61698 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:35:51.862382 30311 raft_consensus.cc:482] T 1e40c426f2a847c0a98bf0b666896bd6 P 74dd0463f0e346b8a2873f1911b61698 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:35:51.862417 30311 raft_consensus.cc:3037] T 1e40c426f2a847c0a98bf0b666896bd6 P 74dd0463f0e346b8a2873f1911b61698 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:51.863646 30311 raft_consensus.cc:504] T 1e40c426f2a847c0a98bf0b666896bd6 P 74dd0463f0e346b8a2873f1911b61698 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74dd0463f0e346b8a2873f1911b61698" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 41695 } } peers { permanent_uuid: "a513449bc7e74ab896d04e547cf5f5c4" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35367 } } peers { permanent_uuid: "c85caf3fc6b841f083b08f47ff8c17fc" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 38271 } }
I20260928 06:35:51.863796 30311 leader_election.cc:288] T 1e40c426f2a847c0a98bf0b666896bd6 P 74dd0463f0e346b8a2873f1911b61698 [CANDIDATE]: Term 1 election: Requested vote from peers a513449bc7e74ab896d04e547cf5f5c4 (127.16.157.67:35367), c85caf3fc6b841f083b08f47ff8c17fc (127.16.157.66:38271)
I20260928 06:35:51.864084 30080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e40c426f2a847c0a98bf0b666896bd6" candidate_uuid: "74dd0463f0e346b8a2873f1911b61698" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c85caf3fc6b841f083b08f47ff8c17fc"
I20260928 06:35:51.864161 30080 raft_consensus.cc:3037] T 1e40c426f2a847c0a98bf0b666896bd6 P c85caf3fc6b841f083b08f47ff8c17fc [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:51.865002 30080 raft_consensus.cc:2454] T 1e40c426f2a847c0a98bf0b666896bd6 P c85caf3fc6b841f083b08f47ff8c17fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74dd0463f0e346b8a2873f1911b61698 in term 1.
I20260928 06:35:51.865180 29894 leader_election.cc:302] T 1e40c426f2a847c0a98bf0b666896bd6 P 74dd0463f0e346b8a2873f1911b61698 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74dd0463f0e346b8a2873f1911b61698, c85caf3fc6b841f083b08f47ff8c17fc; no voters:
I20260928 06:35:51.865306 30311 raft_consensus.cc:2781] T 1e40c426f2a847c0a98bf0b666896bd6 P 74dd0463f0e346b8a2873f1911b61698 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:51.865437 30311 raft_consensus.cc:686] T 1e40c426f2a847c0a98bf0b666896bd6 P 74dd0463f0e346b8a2873f1911b61698 [term 1 LEADER]: Becoming Leader. State: Replica: 74dd0463f0e346b8a2873f1911b61698, State: Running, Role: LEADER
I20260928 06:35:51.865526 30311 consensus_queue.cc:237] T 1e40c426f2a847c0a98bf0b666896bd6 P 74dd0463f0e346b8a2873f1911b61698 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74dd0463f0e346b8a2873f1911b61698" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 41695 } } peers { permanent_uuid: "a513449bc7e74ab896d04e547cf5f5c4" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35367 } } peers { permanent_uuid: "c85caf3fc6b841f083b08f47ff8c17fc" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 38271 } }
I20260928 06:35:51.866459 29806 catalog_manager.cc:5495] T 1e40c426f2a847c0a98bf0b666896bd6 P 74dd0463f0e346b8a2873f1911b61698 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74dd0463f0e346b8a2873f1911b61698 (127.16.157.65). New cstate: current_term: 1 leader_uuid: "74dd0463f0e346b8a2873f1911b61698" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74dd0463f0e346b8a2873f1911b61698" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 41695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a513449bc7e74ab896d04e547cf5f5c4" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c85caf3fc6b841f083b08f47ff8c17fc" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 38271 } health_report { overall_health: UNKNOWN } } }
I20260928 06:35:51.867121 30324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:51.857602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.67:35367 (local address 127.16.157.65:43481)
0928 06:35:51.861070 (+ 3468us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:51.861082 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:51.861135 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:51.863952 (+ 2817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:51.863954 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:51.863955 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:51.863987 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:51.863989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:51.864852 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:51.864853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:51.866005 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:51.866008 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:51.866899 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:51.866900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:51.866963 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:51.866966 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:35:51.867010 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 06:35:51.867050 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3456,"thread_start_us":86,"threads_started":1}
I20260928 06:35:51.867165 30304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:51.857801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:43481 (local address 127.16.157.67:35367)
0928 06:35:51.857856 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:51.857857 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:51.861133 (+ 3276us) server_negotiation.cc:407] Connection header received
0928 06:35:51.861190 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:51.861191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:51.861203 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:51.861255 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:51.864071 (+ 2816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:51.864756 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:51.866119 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:51.866746 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:51.866800 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:51.866808 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:51.867043 (+ 235us) server_negotiation.cc:299] Negotiation successful
0928 06:35:51.867095 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 06:35:51.867674 30213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e40c426f2a847c0a98bf0b666896bd6" candidate_uuid: "74dd0463f0e346b8a2873f1911b61698" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a513449bc7e74ab896d04e547cf5f5c4" is_pre_election: true
I20260928 06:35:51.867758 30227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e40c426f2a847c0a98bf0b666896bd6" candidate_uuid: "74dd0463f0e346b8a2873f1911b61698" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a513449bc7e74ab896d04e547cf5f5c4"
I20260928 06:35:51.867832 30213 raft_consensus.cc:2454] T 1e40c426f2a847c0a98bf0b666896bd6 P a513449bc7e74ab896d04e547cf5f5c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74dd0463f0e346b8a2873f1911b61698 in term 0.
I20260928 06:35:51.929373 30302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:51.926660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38610 (local address 127.16.157.65:41695)
0928 06:35:51.926703 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:51.926705 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:51.926714 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:35:51.926792 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:51.926793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:51.926810 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:51.926874 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:35:51.927033 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:51.927503 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:51.928614 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:51.928792 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:51.928839 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:51.929024 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:35:51.929122 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:35:51.929180 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:51.929249 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 06:35:51.929266 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:35:51.931268 30080 raft_consensus.cc:1261] T 1e40c426f2a847c0a98bf0b666896bd6 P c85caf3fc6b841f083b08f47ff8c17fc [term 1 FOLLOWER]: Refusing update from remote peer 74dd0463f0e346b8a2873f1911b61698: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:35:51.931577 30227 raft_consensus.cc:3037] T 1e40c426f2a847c0a98bf0b666896bd6 P a513449bc7e74ab896d04e547cf5f5c4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:51.931635 30311 consensus_queue.cc:1035] T 1e40c426f2a847c0a98bf0b666896bd6 P 74dd0463f0e346b8a2873f1911b61698 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c85caf3fc6b841f083b08f47ff8c17fc" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 38271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:35:51.932545 30227 raft_consensus.cc:1261] T 1e40c426f2a847c0a98bf0b666896bd6 P a513449bc7e74ab896d04e547cf5f5c4 [term 1 FOLLOWER]: Refusing update from remote peer 74dd0463f0e346b8a2873f1911b61698: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:35:51.933239 30311 consensus_queue.cc:1035] T 1e40c426f2a847c0a98bf0b666896bd6 P 74dd0463f0e346b8a2873f1911b61698 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a513449bc7e74ab896d04e547cf5f5c4" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 35367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:35:51.937352 29806 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:35:51.937451 29806 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:35:51.938138 29806 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b26988d78de3439d8a7817a65dc7b1f6: Sending DeleteTablet for 3 replicas of tablet 1e40c426f2a847c0a98bf0b666896bd6
I20260928 06:35:51.938449 29931 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e40c426f2a847c0a98bf0b666896bd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:51 UTC) from {username='kudu', principal='kudu/127.16.157.126@KRBTEST.COM'} at 127.0.0.1:38598
I20260928 06:35:51.938465 30057 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e40c426f2a847c0a98bf0b666896bd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:51 UTC) from {username='kudu', principal='kudu/127.16.157.126@KRBTEST.COM'} at 127.0.0.1:33228
I20260928 06:35:51.938787 30338 tablet_replica.cc:330] T 1e40c426f2a847c0a98bf0b666896bd6 P 74dd0463f0e346b8a2873f1911b61698: stopping tablet replica
I20260928 06:35:51.938832 30190 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e40c426f2a847c0a98bf0b666896bd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:51 UTC) from {username='kudu', principal='kudu/127.16.157.126@KRBTEST.COM'} at 127.0.0.1:42106
I20260928 06:35:51.938918 30338 raft_consensus.cc:2229] T 1e40c426f2a847c0a98bf0b666896bd6 P 74dd0463f0e346b8a2873f1911b61698 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:35:51.938997 30338 raft_consensus.cc:2258] T 1e40c426f2a847c0a98bf0b666896bd6 P 74dd0463f0e346b8a2873f1911b61698 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:51.939370 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 29845
I20260928 06:35:51.939610 30338 ts_tablet_manager.cc:1905] T 1e40c426f2a847c0a98bf0b666896bd6 P 74dd0463f0e346b8a2873f1911b61698: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:51.941737 30340 tablet_replica.cc:330] T 1e40c426f2a847c0a98bf0b666896bd6 P a513449bc7e74ab896d04e547cf5f5c4: stopping tablet replica
I20260928 06:35:51.941838 30340 raft_consensus.cc:2229] T 1e40c426f2a847c0a98bf0b666896bd6 P a513449bc7e74ab896d04e547cf5f5c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:35:51.941926 30340 raft_consensus.cc:2258] T 1e40c426f2a847c0a98bf0b666896bd6 P a513449bc7e74ab896d04e547cf5f5c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:51.942065 30339 tablet_replica.cc:330] T 1e40c426f2a847c0a98bf0b666896bd6 P c85caf3fc6b841f083b08f47ff8c17fc: stopping tablet replica
I20260928 06:35:51.942150 30339 raft_consensus.cc:2229] T 1e40c426f2a847c0a98bf0b666896bd6 P c85caf3fc6b841f083b08f47ff8c17fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:35:51.942233 30339 raft_consensus.cc:2258] T 1e40c426f2a847c0a98bf0b666896bd6 P c85caf3fc6b841f083b08f47ff8c17fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:51.942627 30340 ts_tablet_manager.cc:1905] T 1e40c426f2a847c0a98bf0b666896bd6 P a513449bc7e74ab896d04e547cf5f5c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:51.943481 30339 ts_tablet_manager.cc:1905] T 1e40c426f2a847c0a98bf0b666896bd6 P c85caf3fc6b841f083b08f47ff8c17fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:51.944398 30340 ts_tablet_manager.cc:1918] T 1e40c426f2a847c0a98bf0b666896bd6 P a513449bc7e74ab896d04e547cf5f5c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:35:51.944517 30340 log.cc:1196] T 1e40c426f2a847c0a98bf0b666896bd6 P a513449bc7e74ab896d04e547cf5f5c4: Deleting WAL directory at /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-2/wal/wals/1e40c426f2a847c0a98bf0b666896bd6
I20260928 06:35:51.944895 30340 ts_tablet_manager.cc:1939] T 1e40c426f2a847c0a98bf0b666896bd6 P a513449bc7e74ab896d04e547cf5f5c4: Deleting consensus metadata
I20260928 06:35:51.945442 30339 ts_tablet_manager.cc:1918] T 1e40c426f2a847c0a98bf0b666896bd6 P c85caf3fc6b841f083b08f47ff8c17fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:35:51.945428 29794 catalog_manager.cc:4841] TS a513449bc7e74ab896d04e547cf5f5c4 (127.16.157.67:35367): tablet 1e40c426f2a847c0a98bf0b666896bd6 (table test-table [id=a8e523b2d3eb45538aa2894e0a1ee1f7]) successfully deleted
I20260928 06:35:51.945616 30339 log.cc:1196] T 1e40c426f2a847c0a98bf0b666896bd6 P c85caf3fc6b841f083b08f47ff8c17fc: Deleting WAL directory at /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577196946045-17013-0/minicluster-data/ts-1/wal/wals/1e40c426f2a847c0a98bf0b666896bd6
I20260928 06:35:51.945956 30339 ts_tablet_manager.cc:1939] T 1e40c426f2a847c0a98bf0b666896bd6 P c85caf3fc6b841f083b08f47ff8c17fc: Deleting consensus metadata
I20260928 06:35:51.946475 29793 catalog_manager.cc:4841] TS c85caf3fc6b841f083b08f47ff8c17fc (127.16.157.66:38271): tablet 1e40c426f2a847c0a98bf0b666896bd6 (table test-table [id=a8e523b2d3eb45538aa2894e0a1ee1f7]) successfully deleted
W20260928 06:35:51.946708 29792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.157.65:41695 (error 108)
I20260928 06:35:51.946988 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 30009
I20260928 06:35:51.951990 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 30142
I20260928 06:35:51.952175 30305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:51.952019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.65:41695 (local address 127.0.0.1:38624)
0928 06:35:51.952093 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:51.952135 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.65:41695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 06:35:51.954802 29792 catalog_manager.cc:4584] TS 74dd0463f0e346b8a2873f1911b61698 (127.16.157.65:41695): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1e40c426f2a847c0a98bf0b666896bd6: Network error: Client connection negotiation failed: client connection to 127.16.157.65:41695: connect: Connection refused (error 111)
I20260928 06:35:51.958891 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 29774
I20260928 06:35:51.965384 17013 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 06:35:52.309913 17013 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 06:35:52.310029 17013 mini_ranger.cc:66] Stopping Ranger...
I20260928 06:35:52.654706 17013 mini_ranger.cc:68] Stopped Ranger
2026-09-28 06:35:52.654 UTC [28841] LOG: received smart shutdown request
2026-09-28 06:35:52.658 UTC [28841] LOG: background worker "logical replication launcher" (PID 28850) exited with exit code 1
2026-09-28 06:35:52.659 UTC [28845] LOG: shutting down
2026-09-28 06:35:52.819 UTC [28841] LOG: database system is shut down
2026-09-28T06:35:52Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (47254 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:35:52 dist-test-slave-rrj5 krb5kdc[30350](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:35:52 dist-test-slave-rrj5 krb5kdc[30350](info): set up 2 sockets
Sep 28 06:35:52 dist-test-slave-rrj5 krb5kdc[30350](info): commencing operation
krb5kdc: starting...
W20260928 06:35:53.962214 17013 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Sep 28 06:35:54 dist-test-slave-rrj5 krb5kdc[30350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:35:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:35:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.157.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab.
Entry for principal kudu/127.16.157.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab.
I20260928 06:35:54.080629 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:40429
--webserver_interface=127.16.157.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=kudu/127.16.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:34699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.157.126:40429,127.16.157.125:40269,127.16.157.124:46863
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:54.162899 30363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:54.163096 30363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:54.163117 30363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:54.164172 30363 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:54.164204 30363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:54.164219 30363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:54.164232 30363 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:54.164243 30363 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:35:54.165678 30363 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:34699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.157.126:40429,127.16.157.125:40269,127.16.157.124:46863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=kudu/127.16.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:40429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:35:54.165864 30363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:54.166019 30363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:54.170892 30369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:54.170946 30363 server_base.cc:1034] running on GCE node
W20260928 06:35:54.170899 30368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:54.170902 30371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:54.171350 30363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:54.171650 30363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:54.172821 30363 hybrid_clock.cc:648] HybridClock initialized: now 1790577354172814 us; error 44 us; skew 500 ppm
Sep 28 06:35:54 dist-test-slave-rrj5 krb5kdc[30350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577354, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:54.175341 30363 init.cc:377] Logged in from keytab as kudu/127.16.157.126@KRBTEST.COM (short username kudu)
I20260928 06:35:54.175803 30363 webserver.cc:460] Webserver started at http://127.16.157.126:35373/ using document root <none> and password file <none>
I20260928 06:35:54.176023 30363 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:54.176090 30363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:54.176247 30363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:54.177217 30363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/minicluster-data/master-0/data/instance:
uuid: "debf4dfc0f6d489facb36c7b44324251"
format_stamp: "Formatted at 2026-09-28 06:35:54 on dist-test-slave-rrj5"
server_key: "fd0c1e3574d363c3d1483ac0fc5ba35d"
server_key_iv: "d78e9037497481c096a06d7155944e43"
server_key_version: "encryptionkey@0"
I20260928 06:35:54.177636 30363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/minicluster-data/master-0/wal/instance:
uuid: "debf4dfc0f6d489facb36c7b44324251"
format_stamp: "Formatted at 2026-09-28 06:35:54 on dist-test-slave-rrj5"
server_key: "fd0c1e3574d363c3d1483ac0fc5ba35d"
server_key_iv: "d78e9037497481c096a06d7155944e43"
server_key_version: "encryptionkey@0"
I20260928 06:35:54.178956 30363 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:35:54.179661 30378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:54.179841 30363 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:35:54.179941 30363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/minicluster-data/master-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/minicluster-data/master-0/wal
uuid: "debf4dfc0f6d489facb36c7b44324251"
format_stamp: "Formatted at 2026-09-28 06:35:54 on dist-test-slave-rrj5"
server_key: "fd0c1e3574d363c3d1483ac0fc5ba35d"
server_key_iv: "d78e9037497481c096a06d7155944e43"
server_key_version: "encryptionkey@0"
I20260928 06:35:54.180025 30363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:54.194267 30363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:54.196023 30363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:54.196197 30363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:54.200345 30363 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.126:40429
I20260928 06:35:54.200379 30430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.126:40429 every 8 connection(s)
I20260928 06:35:54.200716 30363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
I20260928 06:35:54.201550 30431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:54.203127 30431 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40429 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:35:54.205281 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 30363
I20260928 06:35:54.205413 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/minicluster-data/master-0/wal/instance
I20260928 06:35:54.205634 17013 external_mini_cluster.cc:1428] Setting key d726341f5ef949e9fb6210ead6718977
Sep 28 06:35:54 dist-test-slave-rrj5 krb5kdc[30350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577354, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.126@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.157.125@KRBTEST.COM; defaulting to no policy
I20260928 06:35:54.212790 30433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.203779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40429 (local address 127.0.0.1:50308)
0928 06:35:54.203956 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:54.203993 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:54.204444 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:54.204688 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:54.204690 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:54.204930 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:54.205080 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:54.205083 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.206397 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.206397 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:54.207101 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:54.207103 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.207634 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.207635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:54.207704 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:54.208225 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:54.208233 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:54.209548 (+ 1315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:54.211276 (+ 1728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:54.211283 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:54.211284 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:54.211472 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:54.211724 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:54.211724 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:54.211725 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:54.211813 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:54.212184 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:54.212190 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:54.212249 (+ 59us) client_negotiation.cc:769] Sending connection context
0928 06:35:54.212375 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 06:35:54.212611 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":108,"thread_start_us":60,"threads_started":1}
I20260928 06:35:54.212793 30434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.203920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50308 (local address 127.16.157.126:40429)
0928 06:35:54.204047 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:54.204050 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:54.204077 (+ 27us) server_negotiation.cc:407] Connection header received
0928 06:35:54.204567 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:54.204569 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:54.204585 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:54.204658 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:54.205220 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.206270 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.207197 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.207530 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.207661 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:54.209673 (+ 2012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:54.209680 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:54.209681 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:54.209688 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:54.211176 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:54.211569 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:54.211572 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:54.211572 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:54.211654 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:54.211911 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:54.211911 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:54.211912 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:54.212112 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:54.212204 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:54.212415 (+ 211us) server_negotiation.cc:299] Negotiation successful
0928 06:35:54.212560 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":93,"thread_start_us":66,"threads_started":1}
I20260928 06:35:54.213508 30431 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.157.125" port: 40269 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:35:54.213894 30433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.213772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.125:40269 (local address 127.0.0.1:56034)
0928 06:35:54.213823 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:54.213866 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.125:40269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 06:35:54.214072 30381 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.157.125:40269: connect: Connection refused (error 111)
I20260928 06:35:54.214568 30433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.214471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.125:40269 (local address 127.0.0.1:56044)
0928 06:35:54.214521 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:54.214550 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.125:40269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260928 06:35:54.214685 30431 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.157.125:40269: Network error: Client connection negotiation failed: client connection to 127.16.157.125:40269: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.125.keytab.
Entry for principal kudu/127.16.157.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.125.keytab.
I20260928 06:35:54.228736 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.125:40269
--webserver_interface=127.16.157.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.125.keytab
--principal=kudu/127.16.157.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:34699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.157.126:40429,127.16.157.125:40269,127.16.157.124:46863
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:35:54.263926 30431 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.157.125" port: 40269 } attempt: 1
I20260928 06:35:54.264380 30433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.264231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.125:40269 (local address 127.0.0.1:56046)
0928 06:35:54.264296 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:54.264334 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.125:40269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 06:35:54.264731 30433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.264640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.125:40269 (local address 127.0.0.1:56048)
0928 06:35:54.264684 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:54.264707 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.125:40269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 06:35:54.264823 30431 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.157.125:40269: Network error: Client connection negotiation failed: client connection to 127.16.157.125:40269: connect: Connection refused (error 111)
W20260928 06:35:54.309908 30438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:54.310155 30438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:54.310196 30438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:54.311335 30438 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:54.311406 30438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:54.311430 30438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:54.311450 30438 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:54.311480 30438 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:35:54.312944 30438 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:34699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.157.126:40429,127.16.157.125:40269,127.16.157.124:46863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.125.keytab
--principal=kudu/127.16.157.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.125:40269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.157.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:35:54.313216 30438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:54.313450 30438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:54.318346 30443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:54.318329 30446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:54.318485 30438 server_base.cc:1034] running on GCE node
W20260928 06:35:54.318325 30444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:54.318805 30438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:54.319113 30438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:54.320255 30438 hybrid_clock.cc:648] HybridClock initialized: now 1790577354320238 us; error 40 us; skew 500 ppm
Sep 28 06:35:54 dist-test-slave-rrj5 krb5kdc[30350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577354, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:54.322705 30438 init.cc:377] Logged in from keytab as kudu/127.16.157.125@KRBTEST.COM (short username kudu)
I20260928 06:35:54.323151 30438 webserver.cc:460] Webserver started at http://127.16.157.125:35339/ using document root <none> and password file <none>
I20260928 06:35:54.323324 30438 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:54.323366 30438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:54.323450 30438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:54.324431 30438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/minicluster-data/master-1/data/instance:
uuid: "ecb1953207e14acfae41e3f7da2a6aa7"
format_stamp: "Formatted at 2026-09-28 06:35:54 on dist-test-slave-rrj5"
server_key: "f274409b28aea90b141b72d7f2e43ca5"
server_key_iv: "ee304f2b897ba5d95335cf2c9c0b609f"
server_key_version: "encryptionkey@0"
I20260928 06:35:54.324731 30438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/minicluster-data/master-1/wal/instance:
uuid: "ecb1953207e14acfae41e3f7da2a6aa7"
format_stamp: "Formatted at 2026-09-28 06:35:54 on dist-test-slave-rrj5"
server_key: "f274409b28aea90b141b72d7f2e43ca5"
server_key_iv: "ee304f2b897ba5d95335cf2c9c0b609f"
server_key_version: "encryptionkey@0"
I20260928 06:35:54.326030 30438 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:35:54.326710 30453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:54.326891 30438 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:35:54.326977 30438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/minicluster-data/master-1/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/minicluster-data/master-1/wal
uuid: "ecb1953207e14acfae41e3f7da2a6aa7"
format_stamp: "Formatted at 2026-09-28 06:35:54 on dist-test-slave-rrj5"
server_key: "f274409b28aea90b141b72d7f2e43ca5"
server_key_iv: "ee304f2b897ba5d95335cf2c9c0b609f"
server_key_version: "encryptionkey@0"
I20260928 06:35:54.327050 30438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:54.332978 30431 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.157.125" port: 40269 } attempt: 2
I20260928 06:35:54.333382 30433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.333256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.125:40269 (local address 127.0.0.1:56062)
0928 06:35:54.333315 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:54.333352 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.125:40269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 06:35:54.333770 30433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.333676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.125:40269 (local address 127.0.0.1:56076)
0928 06:35:54.333720 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:54.333745 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.125:40269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 06:35:54.333864 30431 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.157.125:40269: Network error: Client connection negotiation failed: client connection to 127.16.157.125:40269: connect: Connection refused (error 111)
I20260928 06:35:54.342491 30438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:54.344264 30438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:54.344391 30438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:54.348286 30438 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.125:40269
I20260928 06:35:54.348320 30505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.125:40269 every 8 connection(s)
I20260928 06:35:54.348681 30438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/minicluster-data/master-1/data/info.pb
I20260928 06:35:54.349530 30506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:54.351068 30506 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40429 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:35:54.353157 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 30438
I20260928 06:35:54.353310 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/minicluster-data/master-1/wal/instance
I20260928 06:35:54.353489 17013 external_mini_cluster.cc:1428] Setting key d85e6ab1028483213e3158fdd8ce168f
Sep 28 06:35:54 dist-test-slave-rrj5 krb5kdc[30350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577354, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.125@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:35:54.360518 30434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.351770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50322 (local address 127.16.157.126:40429)
0928 06:35:54.351851 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:54.351854 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:54.352020 (+ 166us) server_negotiation.cc:407] Connection header received
0928 06:35:54.352516 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:54.352518 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:54.352533 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:54.352584 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:54.353194 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.353844 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.355018 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.355220 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.355278 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:54.357506 (+ 2228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:54.357507 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:54.357508 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:54.357516 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:54.359050 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:54.359559 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:54.359559 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:54.359560 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:54.359627 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:54.359914 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:54.359914 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:54.359914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:54.360013 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:54.360062 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:54.360392 (+ 330us) server_negotiation.cc:299] Negotiation successful
0928 06:35:54.360456 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 06:35:54.360769 30508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.351689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40429 (local address 127.0.0.1:50322)
0928 06:35:54.351903 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:54.351938 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:54.352399 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:54.352654 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:54.352657 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:54.352921 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:54.353096 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:54.353099 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.353975 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.353976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:54.354910 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:54.354913 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.355303 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.355304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:54.355458 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:54.356002 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:54.356010 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:54.357408 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:54.359138 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:54.359146 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:54.359148 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:54.359476 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:54.359692 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:54.359693 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:54.359693 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:54.359846 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:54.360083 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:54.360090 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:54.360252 (+ 162us) client_negotiation.cc:769] Sending connection context
0928 06:35:54.360376 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 06:35:54.360542 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":87,"threads_started":1}
I20260928 06:35:54.361505 30506 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.157.125" port: 40269 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.16.157.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.124@KRBTEST.COM" created.
Sep 28 06:35:54 dist-test-slave-rrj5 krb5kdc[30350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577354, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.125@KRBTEST.COM for kudu/127.16.157.125@KRBTEST.COM
I20260928 06:35:54.368865 30510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.361871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56080 (local address 127.16.157.125:40269)
0928 06:35:54.362031 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:54.362035 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:54.362058 (+ 23us) server_negotiation.cc:407] Connection header received
0928 06:35:54.362074 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:54.362076 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:54.362085 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:54.362162 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:54.362438 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.363087 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.363727 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.364024 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.364101 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:54.365946 (+ 1845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:54.365953 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:54.365954 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:54.365960 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:54.367691 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:54.368040 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:54.368043 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:54.368044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:54.368102 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:54.368299 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:54.368299 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:54.368300 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:54.368465 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:54.368527 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:54.368724 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 06:35:54.368793 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":81,"threads_started":1}
I20260928 06:35:54.369037 30508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.361811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.125:40269 (local address 127.0.0.1:56080)
0928 06:35:54.361863 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:54.361873 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:54.361920 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:54.362162 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:54.362163 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:54.362310 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:54.362354 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:54.362355 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.363174 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.363174 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:54.363648 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:54.363649 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.364096 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.364097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:54.364165 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:54.364627 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:54.364631 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:54.365849 (+ 1218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:54.367789 (+ 1940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:54.367790 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:54.367790 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:54.367950 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:54.368167 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:54.368168 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:54.368168 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:54.368221 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:54.368545 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:54.368547 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:54.368602 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 06:35:54.368704 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 06:35:54.368895 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":69}
I20260928 06:35:54.369764 30506 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.157.124" port: 46863 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:35:54.370086 30508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.369979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.124:46863 (local address 127.0.0.1:40564)
0928 06:35:54.370027 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:54.370069 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.124:46863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 06:35:54.370244 30456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.157.124:46863: connect: Connection refused (error 111)
I20260928 06:35:54.370638 30508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.370576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.124:46863 (local address 127.0.0.1:40566)
0928 06:35:54.370601 (+ 25us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:54.370624 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.124:46863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260928 06:35:54.370769 30506 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.157.124:46863: Network error: Client connection negotiation failed: client connection to 127.16.157.124:46863: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.124.keytab.
Entry for principal kudu/127.16.157.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.124.keytab.
I20260928 06:35:54.379827 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.124:46863
--webserver_interface=127.16.157.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.124.keytab
--principal=kudu/127.16.157.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:34699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.157.126:40429,127.16.157.125:40269,127.16.157.124:46863
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:35:54.420100 30506 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.157.124" port: 46863 } attempt: 1
I20260928 06:35:54.420562 30508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.420433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.124:46863 (local address 127.0.0.1:40574)
0928 06:35:54.420495 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:54.420534 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.124:46863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 06:35:54.420887 30508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.420789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.124:46863 (local address 127.0.0.1:40586)
0928 06:35:54.420834 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:54.420858 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.124:46863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 06:35:54.420972 30506 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.157.124:46863: Network error: Client connection negotiation failed: client connection to 127.16.157.124:46863: connect: Connection refused (error 111)
I20260928 06:35:54.424988 30431 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.157.125" port: 40269 } attempt: 3
Sep 28 06:35:54 dist-test-slave-rrj5 krb5kdc[30350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577354, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.126@KRBTEST.COM for kudu/127.16.157.125@KRBTEST.COM
I20260928 06:35:54.432149 30510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.425326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56084 (local address 127.16.157.125:40269)
0928 06:35:54.425396 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:54.425398 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:54.425409 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:35:54.425457 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:54.425458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:54.425469 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:54.425545 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:54.425890 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.426401 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.427126 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.427347 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.427423 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:54.429392 (+ 1969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:54.429393 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:54.429394 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:54.429401 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:54.431026 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:54.431390 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:54.431391 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:54.431391 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:54.431459 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:54.431659 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:54.431659 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:54.431660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:54.431774 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:54.431847 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:54.431993 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 06:35:54.432065 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 06:35:54.432154 30433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.425268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.125:40269 (local address 127.0.0.1:56084)
0928 06:35:54.425307 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:54.425318 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:54.425395 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:54.425564 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:54.425565 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:54.425739 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:54.425809 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:54.425811 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.426492 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.426493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:54.427045 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:54.427047 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.427434 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.427435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:54.427510 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:54.427971 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:54.427975 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:54.429306 (+ 1331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:54.431117 (+ 1811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:54.431118 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:54.431118 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:54.431289 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:54.431524 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:54.431525 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:54.431525 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:54.431587 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:54.431838 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:54.431839 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:54.431883 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 06:35:54.431969 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 06:35:54.432062 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260928 06:35:54.432623 30431 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.157.124" port: 46863 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:35:54.432958 30433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.432875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.124:46863 (local address 127.0.0.1:40596)
0928 06:35:54.432914 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:54.432941 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.124:46863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 06:35:54.433240 30433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.433172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.124:46863 (local address 127.0.0.1:40600)
0928 06:35:54.433200 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:54.433222 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.124:46863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260928 06:35:54.433331 30431 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.157.124:46863: Network error: Client connection negotiation failed: client connection to 127.16.157.124:46863: connect: Connection refused (error 111)
W20260928 06:35:54.462545 30513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:54.462752 30513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:54.462774 30513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:54.463912 30513 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:54.463953 30513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:54.463967 30513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:54.463980 30513 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:54.464030 30513 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:35:54.464566 30431 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.157.124" port: 46863 } attempt: 1
I20260928 06:35:54.465049 30433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.464885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.124:46863 (local address 127.0.0.1:40606)
0928 06:35:54.464971 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:54.465015 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.124:46863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 06:35:54.465435 30433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.465328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.124:46863 (local address 127.0.0.1:40622)
0928 06:35:54.465387 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:54.465415 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.124:46863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 06:35:54.465564 30431 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.157.124:46863: Network error: Client connection negotiation failed: client connection to 127.16.157.124:46863: connect: Connection refused (error 111)
I20260928 06:35:54.465919 30513 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:34699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.157.126:40429,127.16.157.125:40269,127.16.157.124:46863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.124.keytab
--principal=kudu/127.16.157.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.124:46863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.157.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:35:54.466189 30513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:54.466434 30513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:54.471130 30521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:54.471187 30518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:54.471107 30519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:54.471338 30513 server_base.cc:1034] running on GCE node
I20260928 06:35:54.471580 30513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:54.471849 30513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:54.473049 30513 hybrid_clock.cc:648] HybridClock initialized: now 1790577354473034 us; error 25 us; skew 500 ppm
Sep 28 06:35:54 dist-test-slave-rrj5 krb5kdc[30350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577354, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:54.475477 30513 init.cc:377] Logged in from keytab as kudu/127.16.157.124@KRBTEST.COM (short username kudu)
I20260928 06:35:54.475893 30513 webserver.cc:460] Webserver started at http://127.16.157.124:42403/ using document root <none> and password file <none>
I20260928 06:35:54.476100 30513 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:54.476166 30513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:54.476310 30513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:54.477326 30513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/minicluster-data/master-2/data/instance:
uuid: "c5389a5faa794343a1dd45e6a303ccc5"
format_stamp: "Formatted at 2026-09-28 06:35:54 on dist-test-slave-rrj5"
server_key: "739b9a6fec5356f185d5bdc4545ecb11"
server_key_iv: "490b7a6fae4efadd2cbd0531aa598647"
server_key_version: "encryptionkey@0"
I20260928 06:35:54.477730 30513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/minicluster-data/master-2/wal/instance:
uuid: "c5389a5faa794343a1dd45e6a303ccc5"
format_stamp: "Formatted at 2026-09-28 06:35:54 on dist-test-slave-rrj5"
server_key: "739b9a6fec5356f185d5bdc4545ecb11"
server_key_iv: "490b7a6fae4efadd2cbd0531aa598647"
server_key_version: "encryptionkey@0"
I20260928 06:35:54.479012 30513 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:35:54.479697 30528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:54.479878 30513 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:35:54.479970 30513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/minicluster-data/master-2/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/minicluster-data/master-2/wal
uuid: "c5389a5faa794343a1dd45e6a303ccc5"
format_stamp: "Formatted at 2026-09-28 06:35:54 on dist-test-slave-rrj5"
server_key: "739b9a6fec5356f185d5bdc4545ecb11"
server_key_iv: "490b7a6fae4efadd2cbd0531aa598647"
server_key_version: "encryptionkey@0"
I20260928 06:35:54.480053 30513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:54.489099 30506 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.157.124" port: 46863 } attempt: 2
I20260928 06:35:54.489969 30508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.489755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.124:46863 (local address 127.0.0.1:40628)
0928 06:35:54.489804 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:54.489882 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.124:46863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260928 06:35:54.490382 30508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.490277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.124:46863 (local address 127.0.0.1:40642)
0928 06:35:54.490334 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:54.490361 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.124:46863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 06:35:54.490527 30506 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.157.124:46863: Network error: Client connection negotiation failed: client connection to 127.16.157.124:46863: connect: Connection refused (error 111)
I20260928 06:35:54.503039 30513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:54.504832 30513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:54.504940 30513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:54.508777 30513 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.124:46863
I20260928 06:35:54.508827 30580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.124:46863 every 8 connection(s)
I20260928 06:35:54.509168 30513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/minicluster-data/master-2/data/info.pb
I20260928 06:35:54.510102 30581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:54.511600 30581 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40429 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:35:54.514278 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 30513
I20260928 06:35:54.514394 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577196946045-17013-0/minicluster-data/master-2/wal/instance
I20260928 06:35:54.514609 17013 external_mini_cluster.cc:1428] Setting key 59b1b045c6797cdbafff97ee7e74e13b
Sep 28 06:35:54 dist-test-slave-rrj5 krb5kdc[30350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577354, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.124@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
Sep 28 06:35:54 dist-test-slave-rrj5 krb5kdc[30350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.157.124@KRBTEST.COM
I20260928 06:35:54.521059 30434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.512294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50330 (local address 127.16.157.126:40429)
0928 06:35:54.512341 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:54.512343 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:54.512522 (+ 179us) server_negotiation.cc:407] Connection header received
0928 06:35:54.512945 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:54.512946 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:54.512954 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:54.513007 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:54.513537 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.514043 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.515370 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.515688 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.515790 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:54.518150 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:54.518152 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:54.518152 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:54.518160 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:54.519669 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:54.520074 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:54.520075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:54.520076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:54.520138 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:54.520383 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:54.520384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:54.520384 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:54.520519 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:54.520599 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:54.520884 (+ 285us) server_negotiation.cc:299] Negotiation successful
0928 06:35:54.520968 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:35:54.521296 30583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.512236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40429 (local address 127.0.0.1:50330)
0928 06:35:54.512395 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:54.512434 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:54.512870 (+ 436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:54.513056 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:54.513059 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:54.513279 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:54.513471 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:54.513474 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.514119 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.514120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:54.515279 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:54.515281 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.515789 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.515790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:54.515902 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:54.516366 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:54.516376 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:54.518071 (+ 1695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:54.519753 (+ 1682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:54.519761 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:54.519763 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:54.519993 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:54.520207 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:54.520208 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:54.520208 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:54.520308 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:54.520591 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:54.520598 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:54.520734 (+ 136us) client_negotiation.cc:769] Sending connection context
0928 06:35:54.520836 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 06:35:54.520975 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":86,"threads_started":1}
I20260928 06:35:54.521950 30581 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.157.125" port: 40269 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:35:54.523964 30585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.515001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40648 (local address 127.16.157.124:46863)
0928 06:35:54.515140 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:54.515144 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:54.515199 (+ 55us) server_negotiation.cc:407] Connection header received
0928 06:35:54.515245 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:54.515247 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:54.515266 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:54.515360 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:54.516052 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.517078 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.518126 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.518570 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.518641 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:54.520783 (+ 2142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:54.520792 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:54.520792 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:54.520799 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:54.522753 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:54.523115 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:54.523118 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:54.523118 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:54.523179 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:54.523397 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:54.523398 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:54.523399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:54.523536 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:54.523688 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:54.523779 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 06:35:54.523851 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":70,"threads_started":1}
Sep 28 06:35:54 dist-test-slave-rrj5 krb5kdc[30350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577354, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.124@KRBTEST.COM for kudu/127.16.157.125@KRBTEST.COM
Sep 28 06:35:54 dist-test-slave-rrj5 krb5kdc[30350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:35:54.528553 30583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.522209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.125:40269 (local address 127.0.0.1:56096)
0928 06:35:54.522257 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:54.522267 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:54.522312 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:54.522502 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:54.522502 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:54.522688 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:54.522735 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:54.522736 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.523354 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.523355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:54.523867 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:54.523868 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.524362 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.524363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:54.524433 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:54.524821 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:54.524826 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:54.525954 (+ 1128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:54.527554 (+ 1600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:54.527555 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:54.527555 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:54.527737 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:54.527946 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:54.527947 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:54.527948 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:54.528008 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:54.528261 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:54.528262 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:54.528301 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 06:35:54.528380 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 06:35:54.528468 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 06:35:54.528576 30510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.522295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56096 (local address 127.16.157.125:40269)
0928 06:35:54.522348 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:54.522350 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:54.522359 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:35:54.522383 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:54.522384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:54.522412 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:54.522499 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:54.522815 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.523276 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.524034 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.524288 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.524366 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:54.526040 (+ 1674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:54.526042 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:54.526042 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:54.526049 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:54.527449 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:54.527811 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:54.527812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:54.527812 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:54.527871 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:54.528069 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:54.528070 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:54.528070 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:54.528186 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:54.528256 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:54.528418 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 06:35:54.528489 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 06:35:54.529079 30581 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.157.124" port: 46863 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:35:54.531767 30434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.524654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50342 (local address 127.16.157.126:40429)
0928 06:35:54.524691 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:54.524693 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:54.524700 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:35:54.524739 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:54.524740 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:54.524754 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:54.524807 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:54.525243 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.525793 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.526524 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.526819 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.526891 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:54.529001 (+ 2110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:54.529002 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:54.529003 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:54.529010 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:54.530724 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:54.531052 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:54.531053 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:54.531053 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:54.531100 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:54.531281 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:54.531281 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:54.531281 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:54.531398 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:54.531525 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:54.531608 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 06:35:54.531680 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
Sep 28 06:35:54 dist-test-slave-rrj5 krb5kdc[30350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577354, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.124@KRBTEST.COM for kudu/127.16.157.124@KRBTEST.COM
I20260928 06:35:54.536141 30583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.529363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.124:46863 (local address 127.0.0.1:40662)
0928 06:35:54.529435 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:54.529444 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:54.529492 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:54.529671 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:54.529672 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:54.529823 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:54.529851 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:54.529853 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.530763 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.530764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:54.531496 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:54.531498 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.532090 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.532092 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:54.532153 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:54.532534 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:54.532539 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:54.533708 (+ 1169us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:54.535127 (+ 1419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:54.535128 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:54.535129 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:54.535290 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:54.535489 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:54.535489 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:54.535490 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:54.535542 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:54.535784 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:54.535785 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:54.535821 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 06:35:54.535910 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 06:35:54.536017 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 06:35:54.536194 30585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.529415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40662 (local address 127.16.157.124:46863)
0928 06:35:54.529452 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:54.529454 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:54.529478 (+ 24us) server_negotiation.cc:407] Connection header received
0928 06:35:54.529539 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:54.529540 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:54.529549 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:54.529639 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:54.529954 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.530659 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.531581 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.532012 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.532067 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:54.533819 (+ 1752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:54.533820 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:54.533820 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:54.533827 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:54.535034 (+ 1207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:54.535368 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:54.535369 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:54.535369 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:54.535424 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:54.535614 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:54.535615 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:54.535616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:54.535717 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:54.535791 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:54.535946 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 06:35:54.536064 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":19}
Sep 28 06:35:54 dist-test-slave-rrj5 krb5kdc[30350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.157.125@KRBTEST.COM
I20260928 06:35:54.537513 30581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5389a5faa794343a1dd45e6a303ccc5: Bootstrap starting.
I20260928 06:35:54.538429 30581 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5389a5faa794343a1dd45e6a303ccc5: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:54.538720 30581 log.cc:824] T 00000000000000000000000000000000 P c5389a5faa794343a1dd45e6a303ccc5: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:54.539470 30581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5389a5faa794343a1dd45e6a303ccc5: No bootstrap required, opened a new log
I20260928 06:35:54.539749 30510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.532440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56100 (local address 127.16.157.125:40269)
0928 06:35:54.532477 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:54.532479 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:54.532487 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:35:54.532507 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:54.532508 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:54.532517 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:54.532571 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:54.533068 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.533524 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.534397 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.534689 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.534777 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:54.536960 (+ 2183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:54.536961 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:54.536961 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:54.536969 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:54.538598 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:54.538989 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:54.538991 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:54.538991 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:54.539051 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:54.539255 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:54.539256 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:54.539256 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:54.539364 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:54.539550 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:54.539585 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 06:35:54.539656 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:35:54.540729 30431 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.157.124" port: 46863 } attempt: 2
I20260928 06:35:54.541706 17013 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:35:54.541909 30581 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5389a5faa794343a1dd45e6a303ccc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "debf4dfc0f6d489facb36c7b44324251" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40429 } } peers { permanent_uuid: "ecb1953207e14acfae41e3f7da2a6aa7" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 40269 } } peers { permanent_uuid: "c5389a5faa794343a1dd45e6a303ccc5" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 46863 } }
I20260928 06:35:54.542052 30581 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5389a5faa794343a1dd45e6a303ccc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:54.542083 30581 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5389a5faa794343a1dd45e6a303ccc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5389a5faa794343a1dd45e6a303ccc5, State: Initialized, Role: FOLLOWER
I20260928 06:35:54.542254 30581 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5389a5faa794343a1dd45e6a303ccc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "debf4dfc0f6d489facb36c7b44324251" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40429 } } peers { permanent_uuid: "ecb1953207e14acfae41e3f7da2a6aa7" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 40269 } } peers { permanent_uuid: "c5389a5faa794343a1dd45e6a303ccc5" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 46863 } }
I20260928 06:35:54.542619 30581 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5389a5faa794343a1dd45e6a303ccc5 [sys.catalog]: configured and running, proceeding with master startup.
Sep 28 06:35:54 dist-test-slave-rrj5 krb5kdc[30350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577354, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.126@KRBTEST.COM for kudu/127.16.157.124@KRBTEST.COM
W20260928 06:35:54.547712 30606 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c5389a5faa794343a1dd45e6a303ccc5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:35:54.547824 30606 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:35:54.545451 30513 master_runner.cc:186] Error getting master registration for 127.16.157.126:40429: OK, instance_id { permanent_uuid: "debf4dfc0f6d489facb36c7b44324251" instance_seqno: 1790577354199153 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 06:35:54.549491 30513 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 06:35:54.551532 30433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.541002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.124:46863 (local address 127.0.0.1:40678)
0928 06:35:54.541055 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:54.541066 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:54.541107 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:54.541291 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:54.541292 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:54.541456 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:54.541502 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:54.541504 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.542334 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.542335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:54.543149 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:54.543152 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.543673 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.543675 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:54.543758 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:54.544192 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:54.544198 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:54.545651 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:54.550260 (+ 4609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:54.550261 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:54.550261 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:54.550486 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:54.550729 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:54.550730 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:54.550730 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:54.550791 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:54.551074 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:54.551075 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:54.551119 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 06:35:54.551185 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 06:35:54.551439 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 06:35:54.551739 30585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.541135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40678 (local address 127.16.157.124:46863)
0928 06:35:54.541169 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:54.541170 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:54.541178 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:35:54.541196 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:54.541197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:54.541205 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:54.541280 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:54.541589 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.542254 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.543244 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.543538 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.543620 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:54.547034 (+ 3414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:54.547035 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:54.547035 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:54.547043 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:54.550166 (+ 3123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:54.550592 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:54.550593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:54.550594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:54.550650 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:54.550865 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:54.550866 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:54.550866 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:54.550985 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:54.551045 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:54.551322 (+ 277us) server_negotiation.cc:299] Negotiation successful
0928 06:35:54.551394 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:35:54.552651 30599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.543416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40686 (local address 127.16.157.124:46863)
0928 06:35:54.543537 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:54.543538 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:54.544014 (+ 476us) server_negotiation.cc:407] Connection header received
0928 06:35:54.544120 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:54.544122 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:54.544132 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:54.544185 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:54.545067 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.545795 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.546823 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.547110 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.547169 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:54.548583 (+ 1414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:54.548585 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:54.548585 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:54.548592 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:54.551272 (+ 2680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:54.551652 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:54.551653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:54.551654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:54.551709 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:54.551929 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:54.551930 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:54.551930 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:54.552051 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:54.552144 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:54.552466 (+ 322us) server_negotiation.cc:299] Negotiation successful
0928 06:35:54.552548 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":72,"threads_started":1}
I20260928 06:35:54.542593 30587 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5389a5faa794343a1dd45e6a303ccc5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "debf4dfc0f6d489facb36c7b44324251" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40429 } } peers { permanent_uuid: "ecb1953207e14acfae41e3f7da2a6aa7" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 40269 } } peers { permanent_uuid: "c5389a5faa794343a1dd45e6a303ccc5" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 46863 } } }
I20260928 06:35:54.552719 30587 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5389a5faa794343a1dd45e6a303ccc5 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:35:54.554064 30431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P debf4dfc0f6d489facb36c7b44324251: Bootstrap starting.
I20260928 06:35:54.554250 30434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.543695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50358 (local address 127.16.157.126:40429)
0928 06:35:54.543732 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:54.543734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:54.546923 (+ 3189us) server_negotiation.cc:407] Connection header received
0928 06:35:54.546968 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:54.546969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:54.546979 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:54.547033 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:54.547797 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.548543 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.549526 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.549817 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.549860 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:54.551187 (+ 1327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:54.551188 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:54.551188 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:54.551195 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:54.552977 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:54.553388 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:54.553389 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:54.553389 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:54.553445 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:54.553671 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:54.553672 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:54.553672 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:54.553799 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:54.553865 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:54.554072 (+ 207us) server_negotiation.cc:299] Negotiation successful
0928 06:35:54.554149 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 06:35:54.554790 30431 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P debf4dfc0f6d489facb36c7b44324251: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:54.555085 30431 log.cc:824] T 00000000000000000000000000000000 P debf4dfc0f6d489facb36c7b44324251: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:54.555944 30431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P debf4dfc0f6d489facb36c7b44324251: No bootstrap required, opened a new log
I20260928 06:35:54.558085 30431 raft_consensus.cc:348] T 00000000000000000000000000000000 P debf4dfc0f6d489facb36c7b44324251 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "debf4dfc0f6d489facb36c7b44324251" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40429 } } peers { permanent_uuid: "ecb1953207e14acfae41e3f7da2a6aa7" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 40269 } } peers { permanent_uuid: "c5389a5faa794343a1dd45e6a303ccc5" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 46863 } }
I20260928 06:35:54.558252 30431 raft_consensus.cc:374] T 00000000000000000000000000000000 P debf4dfc0f6d489facb36c7b44324251 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:54.558354 30431 raft_consensus.cc:729] T 00000000000000000000000000000000 P debf4dfc0f6d489facb36c7b44324251 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: debf4dfc0f6d489facb36c7b44324251, State: Initialized, Role: FOLLOWER
I20260928 06:35:54.558547 30431 consensus_queue.cc:260] T 00000000000000000000000000000000 P debf4dfc0f6d489facb36c7b44324251 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "debf4dfc0f6d489facb36c7b44324251" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40429 } } peers { permanent_uuid: "ecb1953207e14acfae41e3f7da2a6aa7" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 40269 } } peers { permanent_uuid: "c5389a5faa794343a1dd45e6a303ccc5" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 46863 } }
I20260928 06:35:54.558950 30609 sys_catalog.cc:455] T 00000000000000000000000000000000 P debf4dfc0f6d489facb36c7b44324251 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "debf4dfc0f6d489facb36c7b44324251" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40429 } } peers { permanent_uuid: "ecb1953207e14acfae41e3f7da2a6aa7" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 40269 } } peers { permanent_uuid: "c5389a5faa794343a1dd45e6a303ccc5" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 46863 } } }
I20260928 06:35:54.559011 30431 sys_catalog.cc:564] T 00000000000000000000000000000000 P debf4dfc0f6d489facb36c7b44324251 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:35:54.559090 30609 sys_catalog.cc:458] T 00000000000000000000000000000000 P debf4dfc0f6d489facb36c7b44324251 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:35:54.560271 30510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.543500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56108 (local address 127.16.157.125:40269)
0928 06:35:54.543532 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:54.543534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:54.543857 (+ 323us) server_negotiation.cc:407] Connection header received
0928 06:35:54.543922 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:54.543923 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:54.543934 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:54.544002 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:54.544638 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.545391 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.546237 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.546467 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.546555 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:54.555260 (+ 8705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:54.555262 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:54.555263 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:54.555271 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:54.558823 (+ 3552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:54.559256 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:54.559258 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:54.559258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:54.559318 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:54.559536 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:54.559537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:54.559537 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:54.559671 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:54.559759 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:54.560086 (+ 327us) server_negotiation.cc:299] Negotiation successful
0928 06:35:54.560161 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
W20260928 06:35:54.561265 30620 catalog_manager.cc:1558] T 00000000000000000000000000000000 P debf4dfc0f6d489facb36c7b44324251: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:35:54.561367 30620 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:35:54.561658 30363 master_runner.cc:186] Error getting master registration for 127.16.157.125:40269: OK, instance_id { permanent_uuid: "ecb1953207e14acfae41e3f7da2a6aa7" instance_seqno: 1790577354347149 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 06:35:54.563480 30363 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 06:35:54.581641 30506 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.157.124" port: 46863 } attempt: 3
Sep 28 06:35:54 dist-test-slave-rrj5 krb5kdc[30350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577354, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.125@KRBTEST.COM for kudu/127.16.157.124@KRBTEST.COM
I20260928 06:35:54.588972 30599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.582153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40694 (local address 127.16.157.124:46863)
0928 06:35:54.582190 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:54.582192 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:54.582201 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:35:54.582221 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:54.582222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:54.582233 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:54.582281 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:54.582737 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.583356 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.584157 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.584389 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.584476 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:54.586341 (+ 1865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:54.586342 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:54.586343 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:54.586361 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:54.587813 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:54.588164 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:54.588165 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:54.588166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:54.588235 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:54.588452 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:54.588453 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:54.588454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:54.588570 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:54.588647 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:54.588803 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 06:35:54.588879 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 06:35:54.588972 30508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.581920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.124:46863 (local address 127.0.0.1:40694)
0928 06:35:54.582028 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:54.582041 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:54.582079 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:54.582339 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:54.582340 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:54.582576 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:54.582646 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:54.582649 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.583463 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.583464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:54.584077 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:54.584079 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.584477 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.584478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:54.584535 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:54.584996 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:54.585001 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:54.586255 (+ 1254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:54.587917 (+ 1662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:54.587918 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:54.587918 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:54.588092 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:54.588308 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:54.588308 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:54.588309 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:54.588391 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:54.588639 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:54.588640 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:54.588684 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 06:35:54.588776 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 06:35:54.588875 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260928 06:35:54.590636 30506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecb1953207e14acfae41e3f7da2a6aa7: Bootstrap starting.
I20260928 06:35:54.591470 30506 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ecb1953207e14acfae41e3f7da2a6aa7: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:54.591794 30506 log.cc:824] T 00000000000000000000000000000000 P ecb1953207e14acfae41e3f7da2a6aa7: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:54.592656 30506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecb1953207e14acfae41e3f7da2a6aa7: No bootstrap required, opened a new log
I20260928 06:35:54.594178 30506 raft_consensus.cc:348] T 00000000000000000000000000000000 P ecb1953207e14acfae41e3f7da2a6aa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "debf4dfc0f6d489facb36c7b44324251" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40429 } } peers { permanent_uuid: "ecb1953207e14acfae41e3f7da2a6aa7" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 40269 } } peers { permanent_uuid: "c5389a5faa794343a1dd45e6a303ccc5" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 46863 } }
I20260928 06:35:54.594321 30506 raft_consensus.cc:374] T 00000000000000000000000000000000 P ecb1953207e14acfae41e3f7da2a6aa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:54.594419 30506 raft_consensus.cc:729] T 00000000000000000000000000000000 P ecb1953207e14acfae41e3f7da2a6aa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecb1953207e14acfae41e3f7da2a6aa7, State: Initialized, Role: FOLLOWER
I20260928 06:35:54.594578 30506 consensus_queue.cc:260] T 00000000000000000000000000000000 P ecb1953207e14acfae41e3f7da2a6aa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "debf4dfc0f6d489facb36c7b44324251" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40429 } } peers { permanent_uuid: "ecb1953207e14acfae41e3f7da2a6aa7" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 40269 } } peers { permanent_uuid: "c5389a5faa794343a1dd45e6a303ccc5" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 46863 } }
I20260928 06:35:54.594851 30623 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecb1953207e14acfae41e3f7da2a6aa7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "debf4dfc0f6d489facb36c7b44324251" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40429 } } peers { permanent_uuid: "ecb1953207e14acfae41e3f7da2a6aa7" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 40269 } } peers { permanent_uuid: "c5389a5faa794343a1dd45e6a303ccc5" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 46863 } } }
I20260928 06:35:54.594909 30506 sys_catalog.cc:564] T 00000000000000000000000000000000 P ecb1953207e14acfae41e3f7da2a6aa7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:35:54.594982 30623 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecb1953207e14acfae41e3f7da2a6aa7 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 06:35:54.596870 30634 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ecb1953207e14acfae41e3f7da2a6aa7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:35:54.596977 30634 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:35:54.599673 30438 master_runner.cc:263] No leader master found from master ecb1953207e14acfae41e3f7da2a6aa7
I20260928 06:35:54.599716 30438 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 06:35:54.688197 30609 raft_consensus.cc:482] T 00000000000000000000000000000000 P debf4dfc0f6d489facb36c7b44324251 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:35:54.688337 30609 raft_consensus.cc:504] T 00000000000000000000000000000000 P debf4dfc0f6d489facb36c7b44324251 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "debf4dfc0f6d489facb36c7b44324251" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40429 } } peers { permanent_uuid: "ecb1953207e14acfae41e3f7da2a6aa7" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 40269 } } peers { permanent_uuid: "c5389a5faa794343a1dd45e6a303ccc5" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 46863 } }
I20260928 06:35:54.688577 30609 leader_election.cc:288] T 00000000000000000000000000000000 P debf4dfc0f6d489facb36c7b44324251 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecb1953207e14acfae41e3f7da2a6aa7 (127.16.157.125:40269), c5389a5faa794343a1dd45e6a303ccc5 (127.16.157.124:46863)
I20260928 06:35:54.689155 30480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "debf4dfc0f6d489facb36c7b44324251" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecb1953207e14acfae41e3f7da2a6aa7" is_pre_election: true
I20260928 06:35:54.689313 30480 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ecb1953207e14acfae41e3f7da2a6aa7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate debf4dfc0f6d489facb36c7b44324251 in term 0.
I20260928 06:35:54.689728 30555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "debf4dfc0f6d489facb36c7b44324251" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5389a5faa794343a1dd45e6a303ccc5" is_pre_election: true
I20260928 06:35:54.689899 30555 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c5389a5faa794343a1dd45e6a303ccc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate debf4dfc0f6d489facb36c7b44324251 in term 0.
I20260928 06:35:54.690246 30381 leader_election.cc:302] T 00000000000000000000000000000000 P debf4dfc0f6d489facb36c7b44324251 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5389a5faa794343a1dd45e6a303ccc5, debf4dfc0f6d489facb36c7b44324251; no voters:
I20260928 06:35:54.690490 30609 raft_consensus.cc:2781] T 00000000000000000000000000000000 P debf4dfc0f6d489facb36c7b44324251 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:35:54.690548 30609 raft_consensus.cc:482] T 00000000000000000000000000000000 P debf4dfc0f6d489facb36c7b44324251 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:35:54.690572 30609 raft_consensus.cc:3037] T 00000000000000000000000000000000 P debf4dfc0f6d489facb36c7b44324251 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:54.691550 30609 raft_consensus.cc:504] T 00000000000000000000000000000000 P debf4dfc0f6d489facb36c7b44324251 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "debf4dfc0f6d489facb36c7b44324251" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40429 } } peers { permanent_uuid: "ecb1953207e14acfae41e3f7da2a6aa7" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 40269 } } peers { permanent_uuid: "c5389a5faa794343a1dd45e6a303ccc5" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 46863 } }
I20260928 06:35:54.691679 30609 leader_election.cc:288] T 00000000000000000000000000000000 P debf4dfc0f6d489facb36c7b44324251 [CANDIDATE]: Term 1 election: Requested vote from peers ecb1953207e14acfae41e3f7da2a6aa7 (127.16.157.125:40269), c5389a5faa794343a1dd45e6a303ccc5 (127.16.157.124:46863)
I20260928 06:35:54.692013 30555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "debf4dfc0f6d489facb36c7b44324251" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5389a5faa794343a1dd45e6a303ccc5"
I20260928 06:35:54.692095 30555 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5389a5faa794343a1dd45e6a303ccc5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:54.693091 30555 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c5389a5faa794343a1dd45e6a303ccc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate debf4dfc0f6d489facb36c7b44324251 in term 1.
I20260928 06:35:54.693308 30480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "debf4dfc0f6d489facb36c7b44324251" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecb1953207e14acfae41e3f7da2a6aa7"
I20260928 06:35:54.693392 30480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ecb1953207e14acfae41e3f7da2a6aa7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:54.694346 30480 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ecb1953207e14acfae41e3f7da2a6aa7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate debf4dfc0f6d489facb36c7b44324251 in term 1.
I20260928 06:35:54.694650 30381 leader_election.cc:302] T 00000000000000000000000000000000 P debf4dfc0f6d489facb36c7b44324251 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: debf4dfc0f6d489facb36c7b44324251, ecb1953207e14acfae41e3f7da2a6aa7; no voters:
I20260928 06:35:54.695765 30609 raft_consensus.cc:2781] T 00000000000000000000000000000000 P debf4dfc0f6d489facb36c7b44324251 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:54.695940 30609 raft_consensus.cc:686] T 00000000000000000000000000000000 P debf4dfc0f6d489facb36c7b44324251 [term 1 LEADER]: Becoming Leader. State: Replica: debf4dfc0f6d489facb36c7b44324251, State: Running, Role: LEADER
I20260928 06:35:54.696033 30609 consensus_queue.cc:237] T 00000000000000000000000000000000 P debf4dfc0f6d489facb36c7b44324251 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "debf4dfc0f6d489facb36c7b44324251" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40429 } } peers { permanent_uuid: "ecb1953207e14acfae41e3f7da2a6aa7" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 40269 } } peers { permanent_uuid: "c5389a5faa794343a1dd45e6a303ccc5" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 46863 } }
I20260928 06:35:54.696465 30609 sys_catalog.cc:455] T 00000000000000000000000000000000 P debf4dfc0f6d489facb36c7b44324251 [sys.catalog]: SysCatalogTable state changed. Reason: New leader debf4dfc0f6d489facb36c7b44324251. Latest consensus state: current_term: 1 leader_uuid: "debf4dfc0f6d489facb36c7b44324251" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "debf4dfc0f6d489facb36c7b44324251" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40429 } } peers { permanent_uuid: "ecb1953207e14acfae41e3f7da2a6aa7" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 40269 } } peers { permanent_uuid: "c5389a5faa794343a1dd45e6a303ccc5" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 46863 } } }
I20260928 06:35:54.696555 30609 sys_catalog.cc:458] T 00000000000000000000000000000000 P debf4dfc0f6d489facb36c7b44324251 [sys.catalog]: This master's current role is: LEADER
W20260928 06:35:54.696971 30384 tablet.cc:2369] T 00000000000000000000000000000000 P debf4dfc0f6d489facb36c7b44324251: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:35:54.697374 30638 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:35:54.698118 30638 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:35:54.699697 30555 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c5389a5faa794343a1dd45e6a303ccc5 [term 1 FOLLOWER]: Refusing update from remote peer debf4dfc0f6d489facb36c7b44324251: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:35:54.700106 30636 consensus_queue.cc:1035] T 00000000000000000000000000000000 P debf4dfc0f6d489facb36c7b44324251 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5389a5faa794343a1dd45e6a303ccc5" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 46863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:35:54.701452 30587 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5389a5faa794343a1dd45e6a303ccc5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader debf4dfc0f6d489facb36c7b44324251. Latest consensus state: current_term: 1 leader_uuid: "debf4dfc0f6d489facb36c7b44324251" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "debf4dfc0f6d489facb36c7b44324251" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40429 } } peers { permanent_uuid: "ecb1953207e14acfae41e3f7da2a6aa7" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 40269 } } peers { permanent_uuid: "c5389a5faa794343a1dd45e6a303ccc5" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 46863 } } }
I20260928 06:35:54.701558 30587 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5389a5faa794343a1dd45e6a303ccc5 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:35:54.702044 30480 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ecb1953207e14acfae41e3f7da2a6aa7 [term 1 FOLLOWER]: Refusing update from remote peer debf4dfc0f6d489facb36c7b44324251: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:35:54.702342 30636 consensus_queue.cc:1035] T 00000000000000000000000000000000 P debf4dfc0f6d489facb36c7b44324251 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecb1953207e14acfae41e3f7da2a6aa7" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 40269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:35:54.703562 30623 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecb1953207e14acfae41e3f7da2a6aa7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader debf4dfc0f6d489facb36c7b44324251. Latest consensus state: current_term: 1 leader_uuid: "debf4dfc0f6d489facb36c7b44324251" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "debf4dfc0f6d489facb36c7b44324251" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40429 } } peers { permanent_uuid: "ecb1953207e14acfae41e3f7da2a6aa7" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 40269 } } peers { permanent_uuid: "c5389a5faa794343a1dd45e6a303ccc5" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 46863 } } }
I20260928 06:35:54.703651 30623 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecb1953207e14acfae41e3f7da2a6aa7 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:35:54.704901 30636 sys_catalog.cc:455] T 00000000000000000000000000000000 P debf4dfc0f6d489facb36c7b44324251 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "debf4dfc0f6d489facb36c7b44324251" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "debf4dfc0f6d489facb36c7b44324251" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40429 } } peers { permanent_uuid: "ecb1953207e14acfae41e3f7da2a6aa7" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 40269 } } peers { permanent_uuid: "c5389a5faa794343a1dd45e6a303ccc5" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 46863 } } }
I20260928 06:35:54.704978 30636 sys_catalog.cc:458] T 00000000000000000000000000000000 P debf4dfc0f6d489facb36c7b44324251 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:54.705475 30623 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecb1953207e14acfae41e3f7da2a6aa7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "debf4dfc0f6d489facb36c7b44324251" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "debf4dfc0f6d489facb36c7b44324251" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40429 } } peers { permanent_uuid: "ecb1953207e14acfae41e3f7da2a6aa7" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 40269 } } peers { permanent_uuid: "c5389a5faa794343a1dd45e6a303ccc5" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 46863 } } }
I20260928 06:35:54.705549 30623 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecb1953207e14acfae41e3f7da2a6aa7 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:35:54.705801 30639 mvcc.cc:205] Tried to move back new op lower bound from 7334204844847747072 to 7334204844835606528. Current Snapshot: MvccSnapshot[applied={T|T < 7334204844847747072}]
I20260928 06:35:54.706000 30642 mvcc.cc:205] Tried to move back new op lower bound from 7334204844847747072 to 7334204844835606528. Current Snapshot: MvccSnapshot[applied={T|T < 7334204844847747072}]
I20260928 06:35:54.706784 30609 sys_catalog.cc:455] T 00000000000000000000000000000000 P debf4dfc0f6d489facb36c7b44324251 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "debf4dfc0f6d489facb36c7b44324251" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "debf4dfc0f6d489facb36c7b44324251" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40429 } } peers { permanent_uuid: "ecb1953207e14acfae41e3f7da2a6aa7" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 40269 } } peers { permanent_uuid: "c5389a5faa794343a1dd45e6a303ccc5" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 46863 } } }
I20260928 06:35:54.706858 30609 sys_catalog.cc:458] T 00000000000000000000000000000000 P debf4dfc0f6d489facb36c7b44324251 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:54.707226 30587 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5389a5faa794343a1dd45e6a303ccc5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "debf4dfc0f6d489facb36c7b44324251" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "debf4dfc0f6d489facb36c7b44324251" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40429 } } peers { permanent_uuid: "ecb1953207e14acfae41e3f7da2a6aa7" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 40269 } } peers { permanent_uuid: "c5389a5faa794343a1dd45e6a303ccc5" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 46863 } } }
I20260928 06:35:54.707309 30587 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5389a5faa794343a1dd45e6a303ccc5 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:35:54.708105 30640 mvcc.cc:205] Tried to move back new op lower bound from 7334204844847747072 to 7334204844835606528. Current Snapshot: MvccSnapshot[applied={T|T < 7334204844847747072}]
I20260928 06:35:54.708696 30638 catalog_manager.cc:1347] Generated new cluster ID: 9d23ebf6356a4fd4ab0a42939a74cb0e
I20260928 06:35:54.708755 30638 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:35:54.728837 30638 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:35:54.729785 30638 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:35:54.736308 30638 catalog_manager.cc:5868] T 00000000000000000000000000000000 P debf4dfc0f6d489facb36c7b44324251: Generated new TSK 0
I20260928 06:35:54.736541 30638 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:35:55.548353 30606 catalog_manager.cc:1259] Loaded cluster ID: 9d23ebf6356a4fd4ab0a42939a74cb0e
I20260928 06:35:55.548470 30606 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c5389a5faa794343a1dd45e6a303ccc5: loading cluster ID for follower catalog manager: success
I20260928 06:35:55.549302 30606 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c5389a5faa794343a1dd45e6a303ccc5: acquiring CA information for follower catalog manager: success
I20260928 06:35:55.549642 30606 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c5389a5faa794343a1dd45e6a303ccc5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 06:35:55.597436 30634 catalog_manager.cc:1259] Loaded cluster ID: 9d23ebf6356a4fd4ab0a42939a74cb0e
I20260928 06:35:55.597517 30634 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ecb1953207e14acfae41e3f7da2a6aa7: loading cluster ID for follower catalog manager: success
I20260928 06:35:55.598379 30634 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ecb1953207e14acfae41e3f7da2a6aa7: acquiring CA information for follower catalog manager: success
I20260928 06:35:55.598706 30634 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ecb1953207e14acfae41e3f7da2a6aa7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 06:35:55.799490 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 30363
I20260928 06:35:55.805001 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 30438
I20260928 06:35:55.810092 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 30513
2026-09-28T06:35:55Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2872 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577196946045-17013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:35:55 dist-test-slave-rrj5 krb5kdc[30652](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:35:55 dist-test-slave-rrj5 krb5kdc[30652](info): set up 2 sockets
Sep 28 06:35:55 dist-test-slave-rrj5 krb5kdc[30652](info): commencing operation
krb5kdc: starting...
W20260928 06:35:56.842469 17013 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Sep 28 06:35:56 dist-test-slave-rrj5 krb5kdc[30652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:35:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:35:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.157.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab.
Entry for principal kudu/127.16.157.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab.
I20260928 06:35:56.961870 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577196946045-17013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577196946045-17013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577196946045-17013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:33779
--webserver_interface=127.16.157.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=kudu/127.16.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:41995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.157.126:33779
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:57.041468 30665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:57.041693 30665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:57.041718 30665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:57.042930 30665 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:57.042973 30665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:57.042989 30665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:57.043001 30665 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:57.043049 30665 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:35:57.044494 30665 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:41995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577196946045-17013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577196946045-17013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.157.126:33779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=kudu/127.16.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:33779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577196946045-17013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:35:57.044754 30665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:57.044989 30665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:57.050249 30671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:57.050213 30670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:57.050450 30665 server_base.cc:1034] running on GCE node
W20260928 06:35:57.050573 30673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:57.050887 30665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:57.051191 30665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:57.052361 30665 hybrid_clock.cc:648] HybridClock initialized: now 1790577357052350 us; error 41 us; skew 500 ppm
Sep 28 06:35:57 dist-test-slave-rrj5 krb5kdc[30652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577357, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:57.054952 30665 init.cc:377] Logged in from keytab as kudu/127.16.157.126@KRBTEST.COM (short username kudu)
I20260928 06:35:57.055419 30665 webserver.cc:460] Webserver started at http://127.16.157.126:37809/ using document root <none> and password file <none>
I20260928 06:35:57.055660 30665 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:57.055727 30665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:57.055889 30665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:57.056994 30665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577196946045-17013-0/minicluster-data/master-0/data/instance:
uuid: "ed6baf021162436cafe55d909cb778c6"
format_stamp: "Formatted at 2026-09-28 06:35:57 on dist-test-slave-rrj5"
server_key: "4fbe4769ff9be517e99e30d4a70edfd1"
server_key_iv: "66982034860facb184469534c3e43ba6"
server_key_version: "encryptionkey@0"
I20260928 06:35:57.057348 30665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577196946045-17013-0/minicluster-data/master-0/wal/instance:
uuid: "ed6baf021162436cafe55d909cb778c6"
format_stamp: "Formatted at 2026-09-28 06:35:57 on dist-test-slave-rrj5"
server_key: "4fbe4769ff9be517e99e30d4a70edfd1"
server_key_iv: "66982034860facb184469534c3e43ba6"
server_key_version: "encryptionkey@0"
I20260928 06:35:57.058754 30665 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 06:35:57.059499 30680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:57.059682 30665 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:35:57.059783 30665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577196946045-17013-0/minicluster-data/master-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577196946045-17013-0/minicluster-data/master-0/wal
uuid: "ed6baf021162436cafe55d909cb778c6"
format_stamp: "Formatted at 2026-09-28 06:35:57 on dist-test-slave-rrj5"
server_key: "4fbe4769ff9be517e99e30d4a70edfd1"
server_key_iv: "66982034860facb184469534c3e43ba6"
server_key_version: "encryptionkey@0"
I20260928 06:35:57.059876 30665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577196946045-17013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577196946045-17013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577196946045-17013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:57.069839 30665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:57.071614 30665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:57.071779 30665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:57.076167 30665 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.126:33779
I20260928 06:35:57.076256 30732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.126:33779 every 8 connection(s)
I20260928 06:35:57.076570 30665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
I20260928 06:35:57.077379 30733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:57.079573 30733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed6baf021162436cafe55d909cb778c6: Bootstrap starting.
I20260928 06:35:57.080089 30733 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed6baf021162436cafe55d909cb778c6: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:57.080330 30733 log.cc:824] T 00000000000000000000000000000000 P ed6baf021162436cafe55d909cb778c6: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:57.080965 30733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed6baf021162436cafe55d909cb778c6: No bootstrap required, opened a new log
I20260928 06:35:57.082563 30733 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed6baf021162436cafe55d909cb778c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed6baf021162436cafe55d909cb778c6" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 33779 } }
I20260928 06:35:57.082733 30733 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed6baf021162436cafe55d909cb778c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:57.082759 30733 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed6baf021162436cafe55d909cb778c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed6baf021162436cafe55d909cb778c6, State: Initialized, Role: FOLLOWER
I20260928 06:35:57.082858 30733 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed6baf021162436cafe55d909cb778c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed6baf021162436cafe55d909cb778c6" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 33779 } }
I20260928 06:35:57.082937 30733 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed6baf021162436cafe55d909cb778c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:35:57.082973 30733 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed6baf021162436cafe55d909cb778c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:35:57.083045 30733 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed6baf021162436cafe55d909cb778c6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:57.083583 30733 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed6baf021162436cafe55d909cb778c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed6baf021162436cafe55d909cb778c6" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 33779 } }
I20260928 06:35:57.083714 30733 leader_election.cc:302] T 00000000000000000000000000000000 P ed6baf021162436cafe55d909cb778c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed6baf021162436cafe55d909cb778c6; no voters:
I20260928 06:35:57.083895 30733 leader_election.cc:288] T 00000000000000000000000000000000 P ed6baf021162436cafe55d909cb778c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:35:57.083945 30736 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed6baf021162436cafe55d909cb778c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:57.084120 30736 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed6baf021162436cafe55d909cb778c6 [term 1 LEADER]: Becoming Leader. State: Replica: ed6baf021162436cafe55d909cb778c6, State: Running, Role: LEADER
I20260928 06:35:57.084157 30733 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed6baf021162436cafe55d909cb778c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:35:57.084292 30736 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed6baf021162436cafe55d909cb778c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed6baf021162436cafe55d909cb778c6" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 33779 } }
I20260928 06:35:57.084745 30738 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed6baf021162436cafe55d909cb778c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed6baf021162436cafe55d909cb778c6. Latest consensus state: current_term: 1 leader_uuid: "ed6baf021162436cafe55d909cb778c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed6baf021162436cafe55d909cb778c6" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 33779 } } }
I20260928 06:35:57.084833 30738 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed6baf021162436cafe55d909cb778c6 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:57.084865 30737 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed6baf021162436cafe55d909cb778c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed6baf021162436cafe55d909cb778c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed6baf021162436cafe55d909cb778c6" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 33779 } } }
I20260928 06:35:57.084960 30737 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed6baf021162436cafe55d909cb778c6 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:57.085412 30745 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:35:57.086026 30745 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:35:57.086387 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 30665
I20260928 06:35:57.086488 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577196946045-17013-0/minicluster-data/master-0/wal/instance
I20260928 06:35:57.086688 17013 external_mini_cluster.cc:1428] Setting key 65946d43d5b1cf3dc3b41afe8d24f5fb
I20260928 06:35:57.087795 30745 catalog_manager.cc:1347] Generated new cluster ID: 2225c8babdc0498792a8d28ee9abb5b6
I20260928 06:35:57.087847 30745 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:35:57 dist-test-slave-rrj5 krb5kdc[30652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:35:57.097369 30745 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:35:57.097988 30745 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:35:57.098734 30755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:57.087156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55272 (local address 127.16.157.126:33779)
0928 06:35:57.087334 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:57.087343 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:57.087378 (+ 35us) server_negotiation.cc:407] Connection header received
0928 06:35:57.087506 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:57.087512 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:57.087580 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:57.088526 (+ 946us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:57.088618 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:57.090034 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:57.091057 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:57.091674 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:57.091789 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:57.094708 (+ 2919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:57.094712 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:57.094712 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:57.094724 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:57.096712 (+ 1988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:57.097153 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:57.097157 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:57.097157 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:57.097285 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:57.097549 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:57.097550 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:57.097550 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:57.097872 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:57.098027 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:57.098370 (+ 343us) server_negotiation.cc:299] Negotiation successful
0928 06:35:57.098513 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":95,"threads_started":1}
I20260928 06:35:57.099702 17013 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:35:57.109872 30745 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed6baf021162436cafe55d909cb778c6: Generated new TSK 0
I20260928 06:35:57.110088 30745 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:35:57.154182 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 30665
2026-09-28T06:35:57Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1344 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:35:57 dist-test-slave-rrj5 krb5kdc[30760](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:35:57 dist-test-slave-rrj5 krb5kdc[30760](info): set up 2 sockets
Sep 28 06:35:57 dist-test-slave-rrj5 krb5kdc[30760](info): commencing operation
krb5kdc: starting...
W20260928 06:35:58.182228 17013 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Sep 28 06:35:58 dist-test-slave-rrj5 krb5kdc[30760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:35:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:35:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.157.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab.
Entry for principal kudu/127.16.157.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab.
I20260928 06:35:58.300226 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:43889
--webserver_interface=127.16.157.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=kudu/127.16.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:37571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.157.126:43889
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:58.378989 30773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:58.379190 30773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:58.379215 30773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:58.380307 30773 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:58.380347 30773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:58.380360 30773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:58.380373 30773 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:58.380426 30773 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:35:58.381830 30773 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:37571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.157.126:43889
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=kudu/127.16.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:43889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30773
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:35:58.382047 30773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:58.382277 30773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:58.387086 30778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:58.387166 30781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:58.387243 30779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:58.387471 30773 server_base.cc:1034] running on GCE node
I20260928 06:35:58.387759 30773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:58.388032 30773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:58.389200 30773 hybrid_clock.cc:648] HybridClock initialized: now 1790577358389188 us; error 32 us; skew 500 ppm
Sep 28 06:35:58 dist-test-slave-rrj5 krb5kdc[30760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577358, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:58.395594 30773 init.cc:377] Logged in from keytab as kudu/127.16.157.126@KRBTEST.COM (short username kudu)
I20260928 06:35:58.396076 30773 webserver.cc:460] Webserver started at http://127.16.157.126:42253/ using document root <none> and password file <none>
I20260928 06:35:58.396311 30773 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:58.396375 30773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:58.396521 30773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:58.397564 30773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/master-0/data/instance:
uuid: "ab2320c32a4546c7add6995d595eaec1"
format_stamp: "Formatted at 2026-09-28 06:35:58 on dist-test-slave-rrj5"
server_key: "9ff8de3d86ed1367a1bfc03942f0e033"
server_key_iv: "ffe5bd95634356bb1a41af4f7722e39c"
server_key_version: "encryptionkey@0"
I20260928 06:35:58.397984 30773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/master-0/wal/instance:
uuid: "ab2320c32a4546c7add6995d595eaec1"
format_stamp: "Formatted at 2026-09-28 06:35:58 on dist-test-slave-rrj5"
server_key: "9ff8de3d86ed1367a1bfc03942f0e033"
server_key_iv: "ffe5bd95634356bb1a41af4f7722e39c"
server_key_version: "encryptionkey@0"
I20260928 06:35:58.399298 30773 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:35:58.400032 30788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:58.400251 30773 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:35:58.400362 30773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/master-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/master-0/wal
uuid: "ab2320c32a4546c7add6995d595eaec1"
format_stamp: "Formatted at 2026-09-28 06:35:58 on dist-test-slave-rrj5"
server_key: "9ff8de3d86ed1367a1bfc03942f0e033"
server_key_iv: "ffe5bd95634356bb1a41af4f7722e39c"
server_key_version: "encryptionkey@0"
I20260928 06:35:58.400444 30773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:58.411794 30773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:58.413993 30773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:58.414145 30773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:58.418079 30773 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.126:43889
I20260928 06:35:58.418121 30840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.126:43889 every 8 connection(s)
I20260928 06:35:58.418459 30773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
I20260928 06:35:58.419365 30841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:58.421455 30841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab2320c32a4546c7add6995d595eaec1: Bootstrap starting.
I20260928 06:35:58.422066 30841 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab2320c32a4546c7add6995d595eaec1: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:58.422299 30841 log.cc:824] T 00000000000000000000000000000000 P ab2320c32a4546c7add6995d595eaec1: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:58.422911 30841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab2320c32a4546c7add6995d595eaec1: No bootstrap required, opened a new log
I20260928 06:35:58.424436 30841 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab2320c32a4546c7add6995d595eaec1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab2320c32a4546c7add6995d595eaec1" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 43889 } }
I20260928 06:35:58.424559 30841 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab2320c32a4546c7add6995d595eaec1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:58.424618 30841 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab2320c32a4546c7add6995d595eaec1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab2320c32a4546c7add6995d595eaec1, State: Initialized, Role: FOLLOWER
I20260928 06:35:58.424835 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 30773
I20260928 06:35:58.424815 30841 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab2320c32a4546c7add6995d595eaec1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab2320c32a4546c7add6995d595eaec1" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 43889 } }
I20260928 06:35:58.424896 30841 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab2320c32a4546c7add6995d595eaec1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:35:58.424933 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/master-0/wal/instance
I20260928 06:35:58.424930 30841 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab2320c32a4546c7add6995d595eaec1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:35:58.424979 30841 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab2320c32a4546c7add6995d595eaec1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:58.425139 17013 external_mini_cluster.cc:1428] Setting key b5d2f417acc7394d8b95ea1368daca19
I20260928 06:35:58.426049 30841 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab2320c32a4546c7add6995d595eaec1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab2320c32a4546c7add6995d595eaec1" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 43889 } }
I20260928 06:35:58.426196 30841 leader_election.cc:302] T 00000000000000000000000000000000 P ab2320c32a4546c7add6995d595eaec1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab2320c32a4546c7add6995d595eaec1; no voters:
I20260928 06:35:58.426397 30841 leader_election.cc:288] T 00000000000000000000000000000000 P ab2320c32a4546c7add6995d595eaec1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:35:58.426424 30846 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab2320c32a4546c7add6995d595eaec1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:58.426630 30846 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab2320c32a4546c7add6995d595eaec1 [term 1 LEADER]: Becoming Leader. State: Replica: ab2320c32a4546c7add6995d595eaec1, State: Running, Role: LEADER
I20260928 06:35:58.426668 30841 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab2320c32a4546c7add6995d595eaec1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:35:58.426779 30846 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab2320c32a4546c7add6995d595eaec1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab2320c32a4546c7add6995d595eaec1" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 43889 } }
I20260928 06:35:58.427111 30846 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab2320c32a4546c7add6995d595eaec1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab2320c32a4546c7add6995d595eaec1. Latest consensus state: current_term: 1 leader_uuid: "ab2320c32a4546c7add6995d595eaec1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab2320c32a4546c7add6995d595eaec1" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 43889 } } }
I20260928 06:35:58.427425 30846 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab2320c32a4546c7add6995d595eaec1 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:58.427356 30848 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab2320c32a4546c7add6995d595eaec1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab2320c32a4546c7add6995d595eaec1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab2320c32a4546c7add6995d595eaec1" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 43889 } } }
I20260928 06:35:58.427774 30848 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab2320c32a4546c7add6995d595eaec1 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:58.427800 30854 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:35:58.428380 30854 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:35:58.430009 30854 catalog_manager.cc:1347] Generated new cluster ID: 407bc57bc9364616a89fa3a8874ffe05
I20260928 06:35:58.430063 30854 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:35:58 dist-test-slave-rrj5 krb5kdc[30760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:35:58.436213 30845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:58.425622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47786 (local address 127.16.157.126:43889)
0928 06:35:58.425818 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:58.425827 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:58.425856 (+ 29us) server_negotiation.cc:407] Connection header received
0928 06:35:58.425943 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:58.425948 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:58.425985 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:58.426089 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:58.427004 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:58.428618 (+ 1614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:58.429701 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:58.430242 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:58.430312 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:58.432837 (+ 2525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:58.432854 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:58.432855 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:58.432866 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:58.434637 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:58.435024 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:58.435028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:58.435028 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:58.435137 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:58.435365 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:58.435366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:58.435367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:58.435618 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:58.435783 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:58.435906 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 06:35:58.436047 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":109,"threads_started":1}
I20260928 06:35:58.465067 30854 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:35:58.465833 30854 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:35:58.469738 30854 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ab2320c32a4546c7add6995d595eaec1: Generated new TSK 0
I20260928 06:35:58.469919 30854 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.157.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab.
Entry for principal kudu/127.16.157.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab.
I20260928 06:35:58.498029 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.65:0
--local_ip_for_outbound_sockets=127.16.157.65
--webserver_interface=127.16.157.65
--webserver_port=0
--tserver_master_addrs=127.16.157.126:43889
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab
--principal=kudu/127.16.157.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:37571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:58.582078 30867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:58.582417 30867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:58.582444 30867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:58.583774 30867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:58.583818 30867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:58.583858 30867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.65
I20260928 06:35:58.585340 30867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:37571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab
--principal=kudu/127.16.157.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.157.65
--webserver_port=0
--tserver_master_addrs=127.16.157.126:43889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.65
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:35:58.585613 30867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:58.585855 30867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:58.590984 30873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:58.591162 30867 server_base.cc:1034] running on GCE node
W20260928 06:35:58.590958 30875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:58.590948 30872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:58.591449 30867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:58.591671 30867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:58.592819 30867 hybrid_clock.cc:648] HybridClock initialized: now 1790577358592799 us; error 38 us; skew 500 ppm
Sep 28 06:35:58 dist-test-slave-rrj5 krb5kdc[30760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577358, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:58.595494 30867 init.cc:377] Logged in from keytab as kudu/127.16.157.65@KRBTEST.COM (short username kudu)
I20260928 06:35:58.595924 30867 webserver.cc:460] Webserver started at http://127.16.157.65:44931/ using document root <none> and password file <none>
I20260928 06:35:58.596148 30867 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:58.596192 30867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:58.596334 30867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:58.597383 30867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-0/data/instance:
uuid: "599bfef32939446d93be83406a84d8bf"
format_stamp: "Formatted at 2026-09-28 06:35:58 on dist-test-slave-rrj5"
server_key: "6721edab5f4d3f725bce9616aeb78a79"
server_key_iv: "33e8eca60a1d89a4cce1d8288d2d6e29"
server_key_version: "encryptionkey@0"
I20260928 06:35:58.597774 30867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-0/wal/instance:
uuid: "599bfef32939446d93be83406a84d8bf"
format_stamp: "Formatted at 2026-09-28 06:35:58 on dist-test-slave-rrj5"
server_key: "6721edab5f4d3f725bce9616aeb78a79"
server_key_iv: "33e8eca60a1d89a4cce1d8288d2d6e29"
server_key_version: "encryptionkey@0"
I20260928 06:35:58.599054 30867 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:35:58.599745 30882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:58.599946 30867 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:35:58.600051 30867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-0/wal
uuid: "599bfef32939446d93be83406a84d8bf"
format_stamp: "Formatted at 2026-09-28 06:35:58 on dist-test-slave-rrj5"
server_key: "6721edab5f4d3f725bce9616aeb78a79"
server_key_iv: "33e8eca60a1d89a4cce1d8288d2d6e29"
server_key_version: "encryptionkey@0"
I20260928 06:35:58.600126 30867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:58.607282 30867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:58.609711 30867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:58.609869 30867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:58.610103 30867 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:58.610416 30867 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:58.610486 30867 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:58.610533 30867 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:58.610560 30867 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:58.616845 30867 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.65:41931
I20260928 06:35:58.616945 30995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.65:41931 every 8 connection(s)
I20260928 06:35:58.617291 30867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
I20260928 06:35:58.622903 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 30867
I20260928 06:35:58.623059 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-0/wal/instance
I20260928 06:35:58.623283 17013 external_mini_cluster.cc:1428] Setting key 4d0bc7817567155871e4bc3c849da053
Sep 28 06:35:58 dist-test-slave-rrj5 krb5kdc[30760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577358, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.65@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:35:58.627506 30845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:58.618145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:34513 (local address 127.16.157.126:43889)
0928 06:35:58.618210 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:58.618213 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:58.618394 (+ 181us) server_negotiation.cc:407] Connection header received
0928 06:35:58.618954 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:58.618956 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:58.618968 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:58.619018 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:58.619982 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:58.620484 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:58.621519 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:58.621805 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:58.621891 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:58.624189 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:58.624190 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:58.624191 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:58.624199 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:58.625888 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:58.626439 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:58.626439 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:58.626440 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:58.626492 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:58.626792 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:58.626792 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:58.626792 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:58.626894 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:58.626943 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:58.627273 (+ 330us) server_negotiation.cc:299] Negotiation successful
0928 06:35:58.627343 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 06:35:58.627737 30998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:58.618100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:43889 (local address 127.16.157.65:34513)
0928 06:35:58.618256 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:58.618305 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:58.618834 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:58.619095 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:58.619099 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:58.619670 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:58.619867 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:58.619872 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:58.620585 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:58.620586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:58.621414 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:58.621415 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:58.621902 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:58.621903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:58.622037 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:58.622517 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:58.622525 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:58.624096 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:58.625969 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:58.625977 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:58.625979 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:58.626362 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:58.626558 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:58.626559 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:58.626560 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:58.626720 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:58.626960 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:58.626967 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:58.627138 (+ 171us) client_negotiation.cc:769] Sending connection context
0928 06:35:58.627254 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 06:35:58.627442 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":80,"threads_started":1}
I20260928 06:35:58.628221 30996 heartbeater.cc:347] Connected to a master server at 127.16.157.126:43889
I20260928 06:35:58.628350 30996 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:58.628575 30996 heartbeater.cc:511] Master 127.16.157.126:43889 requested a full tablet report, sending...
I20260928 06:35:58.629129 30805 ts_manager.cc:194] Registered new tserver with Master: 599bfef32939446d93be83406a84d8bf (127.16.157.65:41931)
I20260928 06:35:58.629848 30805 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.157.65@KRBTEST.COM'} at 127.16.157.65:34513
WARNING: no policy specified for kudu/127.16.157.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab.
Entry for principal kudu/127.16.157.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab.
I20260928 06:35:58.651990 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.66:0
--local_ip_for_outbound_sockets=127.16.157.66
--webserver_interface=127.16.157.66
--webserver_port=0
--tserver_master_addrs=127.16.157.126:43889
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab
--principal=kudu/127.16.157.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:37571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:58.735625 31001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:58.735872 31001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:58.735894 31001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:58.737124 31001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:58.737166 31001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:58.737197 31001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.66
I20260928 06:35:58.738682 31001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:37571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab
--principal=kudu/127.16.157.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.157.66
--webserver_port=0
--tserver_master_addrs=127.16.157.126:43889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31001
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.66
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:35:58.738955 31001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:58.739220 31001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:58.744527 31006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:58.744539 31007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:58.744614 31009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:58.744841 31001 server_base.cc:1034] running on GCE node
I20260928 06:35:58.745061 31001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:58.745357 31001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:58.746500 31001 hybrid_clock.cc:648] HybridClock initialized: now 1790577358746463 us; error 58 us; skew 500 ppm
Sep 28 06:35:58 dist-test-slave-rrj5 krb5kdc[30760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577358, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:58.749116 31001 init.cc:377] Logged in from keytab as kudu/127.16.157.66@KRBTEST.COM (short username kudu)
I20260928 06:35:58.749672 31001 webserver.cc:460] Webserver started at http://127.16.157.66:35161/ using document root <none> and password file <none>
I20260928 06:35:58.749904 31001 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:58.749972 31001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:58.750131 31001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:58.751188 31001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-1/data/instance:
uuid: "3640643bc9624931a62e720a75b7a3d9"
format_stamp: "Formatted at 2026-09-28 06:35:58 on dist-test-slave-rrj5"
server_key: "7fb3e8e258f201951cc66ca655d142df"
server_key_iv: "266ec53330d2f74db71d2954d53d0873"
server_key_version: "encryptionkey@0"
I20260928 06:35:58.751572 31001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-1/wal/instance:
uuid: "3640643bc9624931a62e720a75b7a3d9"
format_stamp: "Formatted at 2026-09-28 06:35:58 on dist-test-slave-rrj5"
server_key: "7fb3e8e258f201951cc66ca655d142df"
server_key_iv: "266ec53330d2f74db71d2954d53d0873"
server_key_version: "encryptionkey@0"
I20260928 06:35:58.752964 31001 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:35:58.753764 31016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:58.753940 31001 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:35:58.754041 31001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-1/wal
uuid: "3640643bc9624931a62e720a75b7a3d9"
format_stamp: "Formatted at 2026-09-28 06:35:58 on dist-test-slave-rrj5"
server_key: "7fb3e8e258f201951cc66ca655d142df"
server_key_iv: "266ec53330d2f74db71d2954d53d0873"
server_key_version: "encryptionkey@0"
I20260928 06:35:58.754127 31001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:58.772368 31001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:58.774839 31001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:58.775015 31001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:58.775277 31001 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:58.775606 31001 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:58.775661 31001 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:58.775705 31001 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:58.775729 31001 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:58.782217 31001 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.66:40265
I20260928 06:35:58.782254 31129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.66:40265 every 8 connection(s)
I20260928 06:35:58.782593 31001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
I20260928 06:35:58.787545 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 31001
I20260928 06:35:58.787655 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-1/wal/instance
I20260928 06:35:58.787885 17013 external_mini_cluster.cc:1428] Setting key 5599c2c872d82bbf36ec468c7ffb68f5
Sep 28 06:35:58 dist-test-slave-rrj5 krb5kdc[30760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577358, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.66@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:35:58.792428 30845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:58.783382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.66:56511 (local address 127.16.157.126:43889)
0928 06:35:58.783456 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:58.783458 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:58.783630 (+ 172us) server_negotiation.cc:407] Connection header received
0928 06:35:58.784189 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:58.784190 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:58.784201 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:58.784278 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:58.785236 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:58.785765 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:58.786843 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:58.787046 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:58.787115 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:58.789452 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:58.789454 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:58.789454 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:58.789462 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:58.791005 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:58.791444 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:58.791445 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:58.791446 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:58.791507 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:58.791765 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:58.791766 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:58.791767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:58.791891 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:58.791963 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:58.792250 (+ 287us) server_negotiation.cc:299] Negotiation successful
0928 06:35:58.792339 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 06:35:58.792631 31132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:58.783324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:43889 (local address 127.16.157.66:56511)
0928 06:35:58.783492 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:58.783535 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:58.784089 (+ 554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:58.784335 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:58.784338 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:58.784917 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:58.785116 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:58.785120 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:58.785924 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:58.785925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:58.786753 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:58.786755 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:58.787124 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:58.787124 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:58.787259 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:58.787958 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:58.787970 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:58.789359 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:58.791081 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:58.791087 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:58.791089 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:58.791343 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:58.791582 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:58.791583 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:58.791584 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:58.791692 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:58.791966 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:58.791974 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:58.792107 (+ 133us) client_negotiation.cc:769] Sending connection context
0928 06:35:58.792227 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 06:35:58.792390 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":99,"threads_started":1}
I20260928 06:35:58.793164 31130 heartbeater.cc:347] Connected to a master server at 127.16.157.126:43889
I20260928 06:35:58.793267 31130 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:58.793468 31130 heartbeater.cc:511] Master 127.16.157.126:43889 requested a full tablet report, sending...
I20260928 06:35:58.793905 30805 ts_manager.cc:194] Registered new tserver with Master: 3640643bc9624931a62e720a75b7a3d9 (127.16.157.66:40265)
I20260928 06:35:58.794484 30805 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.157.66@KRBTEST.COM'} at 127.16.157.66:56511
WARNING: no policy specified for kudu/127.16.157.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab.
Entry for principal kudu/127.16.157.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab.
I20260928 06:35:58.816030 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.67:0
--local_ip_for_outbound_sockets=127.16.157.67
--webserver_interface=127.16.157.67
--webserver_port=0
--tserver_master_addrs=127.16.157.126:43889
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab
--principal=kudu/127.16.157.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:37571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:58.906647 31135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:58.906865 31135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:58.906893 31135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:58.908084 31135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:58.908135 31135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:58.908190 31135 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.67
I20260928 06:35:58.909772 31135 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:37571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab
--principal=kudu/127.16.157.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.157.67
--webserver_port=0
--tserver_master_addrs=127.16.157.126:43889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31135
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.67
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:35:58.910022 31135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:58.910286 31135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:58.915791 31141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:58.915839 31135 server_base.cc:1034] running on GCE node
W20260928 06:35:58.915773 31143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:58.915761 31140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:58.916169 31135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:58.916476 31135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:58.917670 31135 hybrid_clock.cc:648] HybridClock initialized: now 1790577358917652 us; error 27 us; skew 500 ppm
Sep 28 06:35:58 dist-test-slave-rrj5 krb5kdc[30760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577358, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:58.920209 31135 init.cc:377] Logged in from keytab as kudu/127.16.157.67@KRBTEST.COM (short username kudu)
I20260928 06:35:58.920730 31135 webserver.cc:460] Webserver started at http://127.16.157.67:39901/ using document root <none> and password file <none>
I20260928 06:35:58.920970 31135 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:58.921041 31135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:58.921183 31135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:58.922353 31135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-2/data/instance:
uuid: "cea63d9ad0874e73af2ef12dbda8feca"
format_stamp: "Formatted at 2026-09-28 06:35:58 on dist-test-slave-rrj5"
server_key: "f6c58ed13093b7fef3f24b32345450fb"
server_key_iv: "4f604774cf6c3ff4d5eb0178f41e3ef5"
server_key_version: "encryptionkey@0"
I20260928 06:35:58.922720 31135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-2/wal/instance:
uuid: "cea63d9ad0874e73af2ef12dbda8feca"
format_stamp: "Formatted at 2026-09-28 06:35:58 on dist-test-slave-rrj5"
server_key: "f6c58ed13093b7fef3f24b32345450fb"
server_key_iv: "4f604774cf6c3ff4d5eb0178f41e3ef5"
server_key_version: "encryptionkey@0"
I20260928 06:35:58.924098 31135 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:35:58.924983 31150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:58.925186 31135 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:35:58.925294 31135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-2/wal
uuid: "cea63d9ad0874e73af2ef12dbda8feca"
format_stamp: "Formatted at 2026-09-28 06:35:58 on dist-test-slave-rrj5"
server_key: "f6c58ed13093b7fef3f24b32345450fb"
server_key_iv: "4f604774cf6c3ff4d5eb0178f41e3ef5"
server_key_version: "encryptionkey@0"
I20260928 06:35:58.925379 31135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:58.939023 31135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:58.941377 31135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:58.941540 31135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:58.941849 31135 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:58.942175 31135 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:58.942231 31135 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:58.942273 31135 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:58.942301 31135 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:58.948524 31135 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.67:42925
I20260928 06:35:58.948580 31263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.67:42925 every 8 connection(s)
I20260928 06:35:58.948930 31135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
Sep 28 06:35:58 dist-test-slave-rrj5 krb5kdc[30760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577358, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.67@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:35:58.957756 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 31135
I20260928 06:35:58.957851 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-2/wal/instance
I20260928 06:35:58.957988 30845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:58.949760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:48979 (local address 127.16.157.126:43889)
0928 06:35:58.949808 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:58.949810 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:58.950023 (+ 213us) server_negotiation.cc:407] Connection header received
0928 06:35:58.950510 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:58.950511 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:58.950521 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:58.950574 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:58.951494 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:58.951988 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:58.953026 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:58.953219 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:58.953286 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:58.955382 (+ 2096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:58.955383 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:58.955383 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:58.955389 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:58.956595 (+ 1206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:58.957039 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:58.957039 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:58.957040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:58.957091 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:58.957309 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:58.957310 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:58.957310 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:58.957413 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:58.957466 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:58.957788 (+ 322us) server_negotiation.cc:299] Negotiation successful
0928 06:35:58.957896 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:35:58.958025 17013 external_mini_cluster.cc:1428] Setting key dcefa4fb1ab99dd4d9d861181e7e7ad1
I20260928 06:35:58.958163 31266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:58.949735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:43889 (local address 127.16.157.67:48979)
0928 06:35:58.949903 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:58.949941 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:58.950426 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:58.950639 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:58.950644 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:58.951220 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:58.951419 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:58.951424 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:58.952110 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:58.952111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:58.952920 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:58.952922 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:58.953303 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:58.953304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:58.953403 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:58.953958 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:58.953966 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:58.955299 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:58.956667 (+ 1368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:58.956674 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:58.956675 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:58.956954 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:58.957149 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:58.957150 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:58.957150 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:58.957251 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:58.957469 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:58.957474 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:58.957642 (+ 168us) client_negotiation.cc:769] Sending connection context
0928 06:35:58.957738 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 06:35:58.957904 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":93,"threads_started":1}
I20260928 06:35:58.958750 31264 heartbeater.cc:347] Connected to a master server at 127.16.157.126:43889
I20260928 06:35:58.958853 31264 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:58.959052 31264 heartbeater.cc:511] Master 127.16.157.126:43889 requested a full tablet report, sending...
I20260928 06:35:58.959393 30805 ts_manager.cc:194] Registered new tserver with Master: cea63d9ad0874e73af2ef12dbda8feca (127.16.157.67:42925)
I20260928 06:35:58.959782 30805 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.157.67@KRBTEST.COM'} at 127.16.157.67:48979
I20260928 06:35:58.970229 17013 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:35:58.978186 30845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:58.971601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47788 (local address 127.16.157.126:43889)
0928 06:35:58.971644 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:58.971645 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:58.971800 (+ 155us) server_negotiation.cc:407] Connection header received
0928 06:35:58.971878 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:58.971879 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:58.971890 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:58.971946 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:58.972510 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:58.973037 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:58.973846 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:58.974046 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:58.974122 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:58.975450 (+ 1328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:58.975451 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:58.975451 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:58.975460 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:58.976764 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:58.977188 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:58.977190 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:58.977190 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:58.977260 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:58.977506 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:58.977507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:58.977508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:58.977736 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:58.977852 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:58.977978 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 06:35:58.978077 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:35:58.979892 30805 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47788:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:35:58.981452 30805 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:35:58.987957 31275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:58.983814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.67:42925 (local address 127.0.0.1:48752)
0928 06:35:58.983938 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:58.983978 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:58.984043 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:58.984436 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:58.984440 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:58.984443 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:58.984526 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:58.984529 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:58.985611 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:58.985612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:58.986714 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:58.986716 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:58.987665 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:58.987666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:58.987748 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:58.987764 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 06:35:58.987819 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 06:35:58.987872 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":80,"threads_started":1}
I20260928 06:35:58.988138 31276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:58.984043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.65:41931 (local address 127.0.0.1:48852)
0928 06:35:58.984493 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:58.984503 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:58.984559 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:58.984791 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:58.984792 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:58.984794 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:58.984830 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:58.984831 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:58.986159 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:58.986160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:58.987111 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:58.987113 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:58.987887 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:58.987888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:58.987962 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:58.987967 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:35:58.988024 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 06:35:58.988068 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"mutex_wait_us":44,"thread_start_us":78,"threads_started":1}
I20260928 06:35:58.988427 31278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:58.984027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48752 (local address 127.16.157.67:42925)
0928 06:35:58.984231 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:58.984236 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:58.984247 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:35:58.984293 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:58.984295 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:58.984313 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:58.984425 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:58.984670 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:58.985492 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:58.986802 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:58.987564 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:58.988187 (+ 623us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:58.988256 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:58.988290 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 06:35:58.988335 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":120,"threads_started":1}
I20260928 06:35:58.988651 31279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:58.984179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48852 (local address 127.16.157.65:41931)
0928 06:35:58.984359 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:58.984363 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:58.984630 (+ 267us) server_negotiation.cc:407] Connection header received
0928 06:35:58.984685 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:58.984687 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:58.984704 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:58.984795 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:58.984889 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:58.986036 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:58.987211 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:58.987800 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:58.987887 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:58.987949 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:58.988401 (+ 452us) server_negotiation.cc:299] Negotiation successful
0928 06:35:58.988466 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":97,"threads_started":1}
I20260928 06:35:58.988760 31277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:58.983883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.66:40265 (local address 127.0.0.1:42442)
0928 06:35:58.984374 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:58.984385 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:58.984448 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:58.984839 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:58.984840 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:58.984843 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:58.984905 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:58.984907 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:58.986283 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:58.986284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:58.987498 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:58.987502 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:58.988488 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:58.988499 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:58.988565 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:58.988569 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:35:58.988628 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 06:35:58.988670 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":100,"threads_started":1}
I20260928 06:35:58.989248 30930 tablet_service.cc:1467] Processing CreateTablet for tablet d76506ad98434625ad6941ca0970bd03 (DEFAULT_TABLE table=test-table [id=8db8dfaddcfc4222ab9e361d3b868dd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:58.989301 31198 tablet_service.cc:1467] Processing CreateTablet for tablet d76506ad98434625ad6941ca0970bd03 (DEFAULT_TABLE table=test-table [id=8db8dfaddcfc4222ab9e361d3b868dd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:58.989534 30930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d76506ad98434625ad6941ca0970bd03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:58.989710 31198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d76506ad98434625ad6941ca0970bd03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:58.989818 31280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:58.984282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42442 (local address 127.16.157.66:40265)
0928 06:35:58.984651 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:58.984656 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:58.984665 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:35:58.984707 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:58.984709 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:58.984725 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:58.984993 (+ 268us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:58.985083 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:58.986164 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:58.987621 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:58.988393 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:58.988480 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:58.988544 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:58.989678 (+ 1134us) server_negotiation.cc:299] Negotiation successful
0928 06:35:58.989740 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":83,"threads_started":1}
I20260928 06:35:58.992112 31281 tablet_bootstrap.cc:492] T d76506ad98434625ad6941ca0970bd03 P 599bfef32939446d93be83406a84d8bf: Bootstrap starting.
I20260928 06:35:58.992561 31282 tablet_bootstrap.cc:492] T d76506ad98434625ad6941ca0970bd03 P cea63d9ad0874e73af2ef12dbda8feca: Bootstrap starting.
I20260928 06:35:58.992789 31281 tablet_bootstrap.cc:654] T d76506ad98434625ad6941ca0970bd03 P 599bfef32939446d93be83406a84d8bf: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:58.993252 31282 tablet_bootstrap.cc:654] T d76506ad98434625ad6941ca0970bd03 P cea63d9ad0874e73af2ef12dbda8feca: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:58.993409 31281 log.cc:824] T d76506ad98434625ad6941ca0970bd03 P 599bfef32939446d93be83406a84d8bf: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:58.993345 31064 tablet_service.cc:1467] Processing CreateTablet for tablet d76506ad98434625ad6941ca0970bd03 (DEFAULT_TABLE table=test-table [id=8db8dfaddcfc4222ab9e361d3b868dd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:58.993703 31064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d76506ad98434625ad6941ca0970bd03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:58.993909 31282 log.cc:824] T d76506ad98434625ad6941ca0970bd03 P cea63d9ad0874e73af2ef12dbda8feca: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:58.994275 31281 tablet_bootstrap.cc:492] T d76506ad98434625ad6941ca0970bd03 P 599bfef32939446d93be83406a84d8bf: No bootstrap required, opened a new log
I20260928 06:35:58.994349 31281 ts_tablet_manager.cc:1397] T d76506ad98434625ad6941ca0970bd03 P 599bfef32939446d93be83406a84d8bf: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 06:35:58.994638 31282 tablet_bootstrap.cc:492] T d76506ad98434625ad6941ca0970bd03 P cea63d9ad0874e73af2ef12dbda8feca: No bootstrap required, opened a new log
I20260928 06:35:58.994725 31282 ts_tablet_manager.cc:1397] T d76506ad98434625ad6941ca0970bd03 P cea63d9ad0874e73af2ef12dbda8feca: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:35:58.996296 31282 raft_consensus.cc:348] T d76506ad98434625ad6941ca0970bd03 P cea63d9ad0874e73af2ef12dbda8feca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea63d9ad0874e73af2ef12dbda8feca" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 42925 } } peers { permanent_uuid: "599bfef32939446d93be83406a84d8bf" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 41931 } } peers { permanent_uuid: "3640643bc9624931a62e720a75b7a3d9" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 40265 } }
I20260928 06:35:58.996459 31282 raft_consensus.cc:374] T d76506ad98434625ad6941ca0970bd03 P cea63d9ad0874e73af2ef12dbda8feca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:58.996527 31282 raft_consensus.cc:729] T d76506ad98434625ad6941ca0970bd03 P cea63d9ad0874e73af2ef12dbda8feca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea63d9ad0874e73af2ef12dbda8feca, State: Initialized, Role: FOLLOWER
I20260928 06:35:58.996374 31281 raft_consensus.cc:348] T d76506ad98434625ad6941ca0970bd03 P 599bfef32939446d93be83406a84d8bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea63d9ad0874e73af2ef12dbda8feca" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 42925 } } peers { permanent_uuid: "599bfef32939446d93be83406a84d8bf" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 41931 } } peers { permanent_uuid: "3640643bc9624931a62e720a75b7a3d9" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 40265 } }
I20260928 06:35:58.996610 31281 raft_consensus.cc:374] T d76506ad98434625ad6941ca0970bd03 P 599bfef32939446d93be83406a84d8bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:58.996661 31281 raft_consensus.cc:729] T d76506ad98434625ad6941ca0970bd03 P 599bfef32939446d93be83406a84d8bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 599bfef32939446d93be83406a84d8bf, State: Initialized, Role: FOLLOWER
I20260928 06:35:58.996688 31282 consensus_queue.cc:260] T d76506ad98434625ad6941ca0970bd03 P cea63d9ad0874e73af2ef12dbda8feca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea63d9ad0874e73af2ef12dbda8feca" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 42925 } } peers { permanent_uuid: "599bfef32939446d93be83406a84d8bf" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 41931 } } peers { permanent_uuid: "3640643bc9624931a62e720a75b7a3d9" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 40265 } }
I20260928 06:35:58.996785 31281 consensus_queue.cc:260] T d76506ad98434625ad6941ca0970bd03 P 599bfef32939446d93be83406a84d8bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea63d9ad0874e73af2ef12dbda8feca" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 42925 } } peers { permanent_uuid: "599bfef32939446d93be83406a84d8bf" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 41931 } } peers { permanent_uuid: "3640643bc9624931a62e720a75b7a3d9" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 40265 } }
I20260928 06:35:58.997020 31282 ts_tablet_manager.cc:1428] T d76506ad98434625ad6941ca0970bd03 P cea63d9ad0874e73af2ef12dbda8feca: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:35:58.997001 31285 tablet_bootstrap.cc:492] T d76506ad98434625ad6941ca0970bd03 P 3640643bc9624931a62e720a75b7a3d9: Bootstrap starting.
I20260928 06:35:58.997087 31281 ts_tablet_manager.cc:1428] T d76506ad98434625ad6941ca0970bd03 P 599bfef32939446d93be83406a84d8bf: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260928 06:35:58.997174 31264 heartbeater.cc:503] Master 127.16.157.126:43889 was elected leader, sending a full tablet report...
I20260928 06:35:58.997215 30996 heartbeater.cc:503] Master 127.16.157.126:43889 was elected leader, sending a full tablet report...
I20260928 06:35:58.997725 31285 tablet_bootstrap.cc:654] T d76506ad98434625ad6941ca0970bd03 P 3640643bc9624931a62e720a75b7a3d9: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:58.998370 31285 log.cc:824] T d76506ad98434625ad6941ca0970bd03 P 3640643bc9624931a62e720a75b7a3d9: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:58.999055 31285 tablet_bootstrap.cc:492] T d76506ad98434625ad6941ca0970bd03 P 3640643bc9624931a62e720a75b7a3d9: No bootstrap required, opened a new log
I20260928 06:35:58.999114 31285 ts_tablet_manager.cc:1397] T d76506ad98434625ad6941ca0970bd03 P 3640643bc9624931a62e720a75b7a3d9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 06:35:59.000638 31285 raft_consensus.cc:348] T d76506ad98434625ad6941ca0970bd03 P 3640643bc9624931a62e720a75b7a3d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea63d9ad0874e73af2ef12dbda8feca" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 42925 } } peers { permanent_uuid: "599bfef32939446d93be83406a84d8bf" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 41931 } } peers { permanent_uuid: "3640643bc9624931a62e720a75b7a3d9" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 40265 } }
I20260928 06:35:59.000778 31285 raft_consensus.cc:374] T d76506ad98434625ad6941ca0970bd03 P 3640643bc9624931a62e720a75b7a3d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:59.000828 31285 raft_consensus.cc:729] T d76506ad98434625ad6941ca0970bd03 P 3640643bc9624931a62e720a75b7a3d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3640643bc9624931a62e720a75b7a3d9, State: Initialized, Role: FOLLOWER
I20260928 06:35:59.000947 31285 consensus_queue.cc:260] T d76506ad98434625ad6941ca0970bd03 P 3640643bc9624931a62e720a75b7a3d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea63d9ad0874e73af2ef12dbda8feca" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 42925 } } peers { permanent_uuid: "599bfef32939446d93be83406a84d8bf" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 41931 } } peers { permanent_uuid: "3640643bc9624931a62e720a75b7a3d9" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 40265 } }
I20260928 06:35:59.001228 31285 ts_tablet_manager.cc:1428] T d76506ad98434625ad6941ca0970bd03 P 3640643bc9624931a62e720a75b7a3d9: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 06:35:59.001374 31130 heartbeater.cc:503] Master 127.16.157.126:43889 was elected leader, sending a full tablet report...
W20260928 06:35:59.033155 31131 tablet.cc:2369] T d76506ad98434625ad6941ca0970bd03 P 3640643bc9624931a62e720a75b7a3d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:35:59.071422 31287 raft_consensus.cc:482] T d76506ad98434625ad6941ca0970bd03 P 599bfef32939446d93be83406a84d8bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:35:59.071611 31287 raft_consensus.cc:504] T d76506ad98434625ad6941ca0970bd03 P 599bfef32939446d93be83406a84d8bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea63d9ad0874e73af2ef12dbda8feca" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 42925 } } peers { permanent_uuid: "599bfef32939446d93be83406a84d8bf" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 41931 } } peers { permanent_uuid: "3640643bc9624931a62e720a75b7a3d9" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 40265 } }
I20260928 06:35:59.072041 31287 leader_election.cc:288] T d76506ad98434625ad6941ca0970bd03 P 599bfef32939446d93be83406a84d8bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea63d9ad0874e73af2ef12dbda8feca (127.16.157.67:42925), 3640643bc9624931a62e720a75b7a3d9 (127.16.157.66:40265)
I20260928 06:35:59.075050 30998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:59.072272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.66:40265 (local address 127.16.157.65:44315)
0928 06:35:59.072327 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:59.072339 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:59.072390 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:59.072586 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:59.072587 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:59.072588 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:59.072666 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:59.072668 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:59.073327 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:59.073328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:59.074144 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:59.074146 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:59.074785 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:59.074786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:59.074871 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:59.074874 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:35:59.074941 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 06:35:59.074982 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 06:35:59.075071 31280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:59.072418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:44315 (local address 127.16.157.66:40265)
0928 06:35:59.072464 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:59.072465 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:59.072473 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:35:59.072507 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:59.072507 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:59.072517 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:59.072581 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:59.072743 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:59.073242 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:59.074255 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:59.074706 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:59.074791 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:59.074801 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:59.074947 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 06:35:59.074987 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 06:35:59.075436 31084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d76506ad98434625ad6941ca0970bd03" candidate_uuid: "599bfef32939446d93be83406a84d8bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3640643bc9624931a62e720a75b7a3d9" is_pre_election: true
I20260928 06:35:59.075539 31278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:59.072390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:52481 (local address 127.16.157.67:42925)
0928 06:35:59.072441 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:59.072443 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:59.072714 (+ 271us) server_negotiation.cc:407] Connection header received
0928 06:35:59.072795 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:59.072796 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:59.072807 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:59.072896 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:59.073098 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:59.073646 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:59.074603 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:59.075017 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:59.075123 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:59.075132 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:59.075421 (+ 289us) server_negotiation.cc:299] Negotiation successful
0928 06:35:59.075473 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 06:35:59.075531 31290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:59.072277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.67:42925 (local address 127.16.157.65:52481)
0928 06:35:59.072625 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:59.072634 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:59.072721 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:59.072900 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:59.072901 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:59.072902 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:59.073009 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:59.073011 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:59.073733 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:59.073734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:59.074474 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:59.074476 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:59.075254 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:59.075255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:59.075336 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:59.075341 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:35:59.075399 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 06:35:59.075437 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"mutex_wait_us":90,"thread_start_us":157,"threads_started":1}
I20260928 06:35:59.075598 31084 raft_consensus.cc:2454] T d76506ad98434625ad6941ca0970bd03 P 3640643bc9624931a62e720a75b7a3d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 599bfef32939446d93be83406a84d8bf in term 0.
I20260928 06:35:59.075855 30883 leader_election.cc:302] T d76506ad98434625ad6941ca0970bd03 P 599bfef32939446d93be83406a84d8bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3640643bc9624931a62e720a75b7a3d9, 599bfef32939446d93be83406a84d8bf; no voters:
I20260928 06:35:59.075963 31218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d76506ad98434625ad6941ca0970bd03" candidate_uuid: "599bfef32939446d93be83406a84d8bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea63d9ad0874e73af2ef12dbda8feca" is_pre_election: true
I20260928 06:35:59.076025 31287 raft_consensus.cc:2781] T d76506ad98434625ad6941ca0970bd03 P 599bfef32939446d93be83406a84d8bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:35:59.076118 31287 raft_consensus.cc:482] T d76506ad98434625ad6941ca0970bd03 P 599bfef32939446d93be83406a84d8bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:35:59.076125 31218 raft_consensus.cc:2454] T d76506ad98434625ad6941ca0970bd03 P cea63d9ad0874e73af2ef12dbda8feca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 599bfef32939446d93be83406a84d8bf in term 0.
I20260928 06:35:59.076159 31287 raft_consensus.cc:3037] T d76506ad98434625ad6941ca0970bd03 P 599bfef32939446d93be83406a84d8bf [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:59.076943 31287 raft_consensus.cc:504] T d76506ad98434625ad6941ca0970bd03 P 599bfef32939446d93be83406a84d8bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea63d9ad0874e73af2ef12dbda8feca" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 42925 } } peers { permanent_uuid: "599bfef32939446d93be83406a84d8bf" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 41931 } } peers { permanent_uuid: "3640643bc9624931a62e720a75b7a3d9" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 40265 } }
I20260928 06:35:59.077109 31287 leader_election.cc:288] T d76506ad98434625ad6941ca0970bd03 P 599bfef32939446d93be83406a84d8bf [CANDIDATE]: Term 1 election: Requested vote from peers cea63d9ad0874e73af2ef12dbda8feca (127.16.157.67:42925), 3640643bc9624931a62e720a75b7a3d9 (127.16.157.66:40265)
I20260928 06:35:59.077243 31218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d76506ad98434625ad6941ca0970bd03" candidate_uuid: "599bfef32939446d93be83406a84d8bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea63d9ad0874e73af2ef12dbda8feca"
I20260928 06:35:59.077306 31084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d76506ad98434625ad6941ca0970bd03" candidate_uuid: "599bfef32939446d93be83406a84d8bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3640643bc9624931a62e720a75b7a3d9"
I20260928 06:35:59.077347 31218 raft_consensus.cc:3037] T d76506ad98434625ad6941ca0970bd03 P cea63d9ad0874e73af2ef12dbda8feca [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:59.077389 31084 raft_consensus.cc:3037] T d76506ad98434625ad6941ca0970bd03 P 3640643bc9624931a62e720a75b7a3d9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:59.078218 31218 raft_consensus.cc:2454] T d76506ad98434625ad6941ca0970bd03 P cea63d9ad0874e73af2ef12dbda8feca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 599bfef32939446d93be83406a84d8bf in term 1.
I20260928 06:35:59.078222 31084 raft_consensus.cc:2454] T d76506ad98434625ad6941ca0970bd03 P 3640643bc9624931a62e720a75b7a3d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 599bfef32939446d93be83406a84d8bf in term 1.
I20260928 06:35:59.078389 30883 leader_election.cc:302] T d76506ad98434625ad6941ca0970bd03 P 599bfef32939446d93be83406a84d8bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3640643bc9624931a62e720a75b7a3d9, 599bfef32939446d93be83406a84d8bf; no voters:
I20260928 06:35:59.078500 31287 raft_consensus.cc:2781] T d76506ad98434625ad6941ca0970bd03 P 599bfef32939446d93be83406a84d8bf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:59.078656 31287 raft_consensus.cc:686] T d76506ad98434625ad6941ca0970bd03 P 599bfef32939446d93be83406a84d8bf [term 1 LEADER]: Becoming Leader. State: Replica: 599bfef32939446d93be83406a84d8bf, State: Running, Role: LEADER
I20260928 06:35:59.078728 31287 consensus_queue.cc:237] T d76506ad98434625ad6941ca0970bd03 P 599bfef32939446d93be83406a84d8bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea63d9ad0874e73af2ef12dbda8feca" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 42925 } } peers { permanent_uuid: "599bfef32939446d93be83406a84d8bf" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 41931 } } peers { permanent_uuid: "3640643bc9624931a62e720a75b7a3d9" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 40265 } }
I20260928 06:35:59.079602 30804 catalog_manager.cc:5495] T d76506ad98434625ad6941ca0970bd03 P 599bfef32939446d93be83406a84d8bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 599bfef32939446d93be83406a84d8bf (127.16.157.65). New cstate: current_term: 1 leader_uuid: "599bfef32939446d93be83406a84d8bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea63d9ad0874e73af2ef12dbda8feca" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 42925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "599bfef32939446d93be83406a84d8bf" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 41931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3640643bc9624931a62e720a75b7a3d9" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 40265 } health_report { overall_health: UNKNOWN } } }
I20260928 06:35:59.103518 31279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:59.100944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48866 (local address 127.16.157.65:41931)
0928 06:35:59.101020 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:59.101022 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:59.101029 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:35:59.101062 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:59.101062 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:59.101079 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:59.101155 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:35:59.101304 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:59.101835 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:59.102672 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:59.102887 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:59.102961 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:59.103155 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:35:59.103272 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:35:59.103367 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:59.103411 (+ 44us) server_negotiation.cc:299] Negotiation successful
0928 06:35:59.103431 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 06:35:59.105347 31218 raft_consensus.cc:1261] T d76506ad98434625ad6941ca0970bd03 P cea63d9ad0874e73af2ef12dbda8feca [term 1 FOLLOWER]: Refusing update from remote peer 599bfef32939446d93be83406a84d8bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:35:59.105377 31084 raft_consensus.cc:1261] T d76506ad98434625ad6941ca0970bd03 P 3640643bc9624931a62e720a75b7a3d9 [term 1 FOLLOWER]: Refusing update from remote peer 599bfef32939446d93be83406a84d8bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:35:59.105718 31287 consensus_queue.cc:1035] T d76506ad98434625ad6941ca0970bd03 P 599bfef32939446d93be83406a84d8bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea63d9ad0874e73af2ef12dbda8feca" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 42925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:35:59.105868 31291 consensus_queue.cc:1035] T d76506ad98434625ad6941ca0970bd03 P 599bfef32939446d93be83406a84d8bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3640643bc9624931a62e720a75b7a3d9" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 40265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:35:59.111147 30805 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:35:59.111245 30805 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:35:59.111994 30805 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ab2320c32a4546c7add6995d595eaec1: Sending DeleteTablet for 3 replicas of tablet d76506ad98434625ad6941ca0970bd03
I20260928 06:35:59.112346 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 30867
I20260928 06:35:59.112383 31198 tablet_service.cc:1514] Processing DeleteTablet for tablet d76506ad98434625ad6941ca0970bd03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:59 UTC) from {username='kudu', principal='kudu/127.16.157.126@KRBTEST.COM'} at 127.0.0.1:48752
I20260928 06:35:59.112895 31064 tablet_service.cc:1514] Processing DeleteTablet for tablet d76506ad98434625ad6941ca0970bd03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:59 UTC) from {username='kudu', principal='kudu/127.16.157.126@KRBTEST.COM'} at 127.0.0.1:42442
I20260928 06:35:59.113150 31305 tablet_replica.cc:330] T d76506ad98434625ad6941ca0970bd03 P 3640643bc9624931a62e720a75b7a3d9: stopping tablet replica
I20260928 06:35:59.113260 31305 raft_consensus.cc:2229] T d76506ad98434625ad6941ca0970bd03 P 3640643bc9624931a62e720a75b7a3d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:35:59.113368 31305 raft_consensus.cc:2258] T d76506ad98434625ad6941ca0970bd03 P 3640643bc9624931a62e720a75b7a3d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:59.113808 31305 ts_tablet_manager.cc:1905] T d76506ad98434625ad6941ca0970bd03 P 3640643bc9624931a62e720a75b7a3d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:59.115846 31305 ts_tablet_manager.cc:1918] T d76506ad98434625ad6941ca0970bd03 P 3640643bc9624931a62e720a75b7a3d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:35:59.115927 31305 log.cc:1196] T d76506ad98434625ad6941ca0970bd03 P 3640643bc9624931a62e720a75b7a3d9: Deleting WAL directory at /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-1/wal/wals/d76506ad98434625ad6941ca0970bd03
I20260928 06:35:59.116194 31305 ts_tablet_manager.cc:1939] T d76506ad98434625ad6941ca0970bd03 P 3640643bc9624931a62e720a75b7a3d9: Deleting consensus metadata
I20260928 06:35:59.116582 30789 catalog_manager.cc:4841] TS 3640643bc9624931a62e720a75b7a3d9 (127.16.157.66:40265): tablet d76506ad98434625ad6941ca0970bd03 (table test-table [id=8db8dfaddcfc4222ab9e361d3b868dd7]) successfully deleted
W20260928 06:35:59.118902 30790 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.157.65:41931 (error 108)
I20260928 06:35:59.118918 31304 tablet_replica.cc:330] T d76506ad98434625ad6941ca0970bd03 P cea63d9ad0874e73af2ef12dbda8feca: stopping tablet replica
I20260928 06:35:59.119055 31304 raft_consensus.cc:2229] T d76506ad98434625ad6941ca0970bd03 P cea63d9ad0874e73af2ef12dbda8feca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:35:59.119099 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 31001
I20260928 06:35:59.119403 31277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:59.119289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.65:41931 (local address 127.0.0.1:48880)
0928 06:35:59.119335 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:59.119365 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.65:41931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 06:35:59.119500 30790 catalog_manager.cc:4584] TS 599bfef32939446d93be83406a84d8bf (127.16.157.65:41931): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d76506ad98434625ad6941ca0970bd03: Network error: Client connection negotiation failed: client connection to 127.16.157.65:41931: connect: Connection refused (error 111)
I20260928 06:35:59.119740 31304 raft_consensus.cc:2258] T d76506ad98434625ad6941ca0970bd03 P cea63d9ad0874e73af2ef12dbda8feca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:59.121377 31304 ts_tablet_manager.cc:1905] T d76506ad98434625ad6941ca0970bd03 P cea63d9ad0874e73af2ef12dbda8feca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:59.122942 31304 ts_tablet_manager.cc:1918] T d76506ad98434625ad6941ca0970bd03 P cea63d9ad0874e73af2ef12dbda8feca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:35:59.123034 31304 log.cc:1196] T d76506ad98434625ad6941ca0970bd03 P cea63d9ad0874e73af2ef12dbda8feca: Deleting WAL directory at /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-2/wal/wals/d76506ad98434625ad6941ca0970bd03
I20260928 06:35:59.123299 31304 ts_tablet_manager.cc:1939] T d76506ad98434625ad6941ca0970bd03 P cea63d9ad0874e73af2ef12dbda8feca: Deleting consensus metadata
I20260928 06:35:59.123708 30790 catalog_manager.cc:4841] TS cea63d9ad0874e73af2ef12dbda8feca (127.16.157.67:42925): tablet d76506ad98434625ad6941ca0970bd03 (table test-table [id=8db8dfaddcfc4222ab9e361d3b868dd7]) successfully deleted
I20260928 06:35:59.124437 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 31135
I20260928 06:35:59.129704 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 30773
I20260928 06:35:59.135074 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:43889
--webserver_interface=127.16.157.126
--webserver_port=42253
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=kudu/127.16.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:37571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.157.126:43889
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:59.214056 31307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:59.214293 31307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:59.214329 31307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:59.215529 31307 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:59.215600 31307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:59.215624 31307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:59.215651 31307 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:59.215682 31307 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:35:59.217113 31307 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:37571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.16.157.126:43889
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=kudu/127.16.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:43889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.157.126
--webserver_port=42253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:35:59.217388 31307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:59.217623 31307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:59.227061 31316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:59.227092 31314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:59.227321 31307 server_base.cc:1034] running on GCE node
W20260928 06:35:59.227065 31313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:59.227615 31307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:59.227949 31307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:59.229135 31307 hybrid_clock.cc:648] HybridClock initialized: now 1790577359229118 us; error 43 us; skew 500 ppm
Sep 28 06:35:59 dist-test-slave-rrj5 krb5kdc[30760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577359, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:59.231654 31307 init.cc:377] Logged in from keytab as kudu/127.16.157.126@KRBTEST.COM (short username kudu)
I20260928 06:35:59.232136 31307 webserver.cc:460] Webserver started at http://127.16.157.126:42253/ using document root <none> and password file <none>
I20260928 06:35:59.232388 31307 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:59.232470 31307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:59.233872 31307 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:35:59.234545 31323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:59.234716 31307 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:35:59.234824 31307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/master-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/master-0/wal
uuid: "ab2320c32a4546c7add6995d595eaec1"
format_stamp: "Formatted at 2026-09-28 06:35:58 on dist-test-slave-rrj5"
server_key: "9ff8de3d86ed1367a1bfc03942f0e033"
server_key_iv: "ffe5bd95634356bb1a41af4f7722e39c"
server_key_version: "encryptionkey@0"
I20260928 06:35:59.235235 31307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:59.249054 31307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:59.250895 31307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:59.251055 31307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:59.255395 31307 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.126:43889
I20260928 06:35:59.255456 31375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.126:43889 every 8 connection(s)
I20260928 06:35:59.255877 31307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
I20260928 06:35:59.257404 31376 sys_catalog.cc:263] Verifying existing consensus state
I20260928 06:35:59.258131 31376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab2320c32a4546c7add6995d595eaec1: Bootstrap starting.
I20260928 06:35:59.259788 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 31307
I20260928 06:35:59.260066 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.65:41931
--local_ip_for_outbound_sockets=127.16.157.65
--tserver_master_addrs=127.16.157.126:43889
--webserver_port=44931
--webserver_interface=127.16.157.65
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab
--principal=kudu/127.16.157.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:37571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:35:59.267248 31376 log.cc:824] T 00000000000000000000000000000000 P ab2320c32a4546c7add6995d595eaec1: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:59.269135 31376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab2320c32a4546c7add6995d595eaec1: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:35:59.269444 31376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab2320c32a4546c7add6995d595eaec1: Bootstrap complete.
I20260928 06:35:59.271981 31376 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab2320c32a4546c7add6995d595eaec1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab2320c32a4546c7add6995d595eaec1" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 43889 } }
I20260928 06:35:59.272226 31376 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab2320c32a4546c7add6995d595eaec1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab2320c32a4546c7add6995d595eaec1, State: Initialized, Role: FOLLOWER
I20260928 06:35:59.272432 31376 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab2320c32a4546c7add6995d595eaec1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab2320c32a4546c7add6995d595eaec1" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 43889 } }
I20260928 06:35:59.272529 31376 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab2320c32a4546c7add6995d595eaec1 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:35:59.272588 31376 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab2320c32a4546c7add6995d595eaec1 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:35:59.272636 31376 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab2320c32a4546c7add6995d595eaec1 [term 1 FOLLOWER]: Advancing to term 2
I20260928 06:35:59.273707 31376 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab2320c32a4546c7add6995d595eaec1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab2320c32a4546c7add6995d595eaec1" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 43889 } }
I20260928 06:35:59.273830 31376 leader_election.cc:302] T 00000000000000000000000000000000 P ab2320c32a4546c7add6995d595eaec1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab2320c32a4546c7add6995d595eaec1; no voters:
I20260928 06:35:59.274032 31376 leader_election.cc:288] T 00000000000000000000000000000000 P ab2320c32a4546c7add6995d595eaec1 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 06:35:59.274068 31380 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab2320c32a4546c7add6995d595eaec1 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 06:35:59.274233 31380 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab2320c32a4546c7add6995d595eaec1 [term 2 LEADER]: Becoming Leader. State: Replica: ab2320c32a4546c7add6995d595eaec1, State: Running, Role: LEADER
I20260928 06:35:59.274389 31376 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab2320c32a4546c7add6995d595eaec1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:35:59.274406 31380 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab2320c32a4546c7add6995d595eaec1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab2320c32a4546c7add6995d595eaec1" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 43889 } }
I20260928 06:35:59.274784 31381 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab2320c32a4546c7add6995d595eaec1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab2320c32a4546c7add6995d595eaec1. Latest consensus state: current_term: 2 leader_uuid: "ab2320c32a4546c7add6995d595eaec1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab2320c32a4546c7add6995d595eaec1" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 43889 } } }
I20260928 06:35:59.274876 31381 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab2320c32a4546c7add6995d595eaec1 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:59.275115 31382 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab2320c32a4546c7add6995d595eaec1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ab2320c32a4546c7add6995d595eaec1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab2320c32a4546c7add6995d595eaec1" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 43889 } } }
I20260928 06:35:59.275190 31382 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab2320c32a4546c7add6995d595eaec1 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:59.275352 31387 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:35:59.276129 31387 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:35:59.276298 31387 catalog_manager.cc:1259] Loaded cluster ID: 407bc57bc9364616a89fa3a8874ffe05
I20260928 06:35:59.276497 31387 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:35:59.276791 31387 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 06:35:59.281754 31387 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 06:35:59.284825 31387 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:35:59.285009 31387 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ab2320c32a4546c7add6995d595eaec1: Loaded TSK: 0
I20260928 06:35:59.285236 31387 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 06:35:59.345396 31378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:59.345692 31378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:59.345741 31378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:59.346940 31378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:59.347008 31378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:59.347052 31378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.65
I20260928 06:35:59.348505 31378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:37571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab
--principal=kudu/127.16.157.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.65:41931
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.157.65
--webserver_port=44931
--tserver_master_addrs=127.16.157.126:43889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31378
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.65
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:35:59.348829 31378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:59.349069 31378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:59.355103 31404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:59.355113 31403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:59.355067 31406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:59.355782 31378 server_base.cc:1034] running on GCE node
I20260928 06:35:59.355991 31378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:59.356263 31378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:59.357432 31378 hybrid_clock.cc:648] HybridClock initialized: now 1790577359357407 us; error 40 us; skew 500 ppm
Sep 28 06:35:59 dist-test-slave-rrj5 krb5kdc[30760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577359, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:59.359975 31378 init.cc:377] Logged in from keytab as kudu/127.16.157.65@KRBTEST.COM (short username kudu)
I20260928 06:35:59.360461 31378 webserver.cc:460] Webserver started at http://127.16.157.65:44931/ using document root <none> and password file <none>
I20260928 06:35:59.360716 31378 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:59.360771 31378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:59.362319 31378 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 06:35:59.363171 31413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:59.363346 31378 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:35:59.363454 31378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-0/wal
uuid: "599bfef32939446d93be83406a84d8bf"
format_stamp: "Formatted at 2026-09-28 06:35:58 on dist-test-slave-rrj5"
server_key: "6721edab5f4d3f725bce9616aeb78a79"
server_key_iv: "33e8eca60a1d89a4cce1d8288d2d6e29"
server_key_version: "encryptionkey@0"
I20260928 06:35:59.363843 31378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:59.376250 31378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:59.378661 31378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:59.378809 31378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:59.379091 31378 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:59.379634 31420 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 06:35:59.381059 31378 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 06:35:59.381110 31378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 06:35:59.381147 31378 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 06:35:59.381933 31378 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 06:35:59.381979 31378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260928 06:35:59.382022 31420 tablet_bootstrap.cc:492] T d76506ad98434625ad6941ca0970bd03 P 599bfef32939446d93be83406a84d8bf: Bootstrap starting.
I20260928 06:35:59.393692 31420 log.cc:824] T d76506ad98434625ad6941ca0970bd03 P 599bfef32939446d93be83406a84d8bf: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:59.394133 31378 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.65:41931
I20260928 06:35:59.394590 31378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
I20260928 06:35:59.395318 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 31378
I20260928 06:35:59.395620 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.66:40265
--local_ip_for_outbound_sockets=127.16.157.66
--tserver_master_addrs=127.16.157.126:43889
--webserver_port=35161
--webserver_interface=127.16.157.66
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab
--principal=kudu/127.16.157.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:37571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:35:59.396401 31420 tablet_bootstrap.cc:492] T d76506ad98434625ad6941ca0970bd03 P 599bfef32939446d93be83406a84d8bf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:35:59.396613 31420 tablet_bootstrap.cc:492] T d76506ad98434625ad6941ca0970bd03 P 599bfef32939446d93be83406a84d8bf: Bootstrap complete.
I20260928 06:35:59.397948 31420 ts_tablet_manager.cc:1397] T d76506ad98434625ad6941ca0970bd03 P 599bfef32939446d93be83406a84d8bf: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.012s
I20260928 06:35:59.399330 31420 raft_consensus.cc:348] T d76506ad98434625ad6941ca0970bd03 P 599bfef32939446d93be83406a84d8bf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea63d9ad0874e73af2ef12dbda8feca" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 42925 } } peers { permanent_uuid: "599bfef32939446d93be83406a84d8bf" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 41931 } } peers { permanent_uuid: "3640643bc9624931a62e720a75b7a3d9" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 40265 } }
I20260928 06:35:59.399574 31420 raft_consensus.cc:729] T d76506ad98434625ad6941ca0970bd03 P 599bfef32939446d93be83406a84d8bf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 599bfef32939446d93be83406a84d8bf, State: Initialized, Role: FOLLOWER
I20260928 06:35:59.401886 31528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.65:41931 every 8 connection(s)
I20260928 06:35:59.402213 31420 consensus_queue.cc:260] T d76506ad98434625ad6941ca0970bd03 P 599bfef32939446d93be83406a84d8bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea63d9ad0874e73af2ef12dbda8feca" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 42925 } } peers { permanent_uuid: "599bfef32939446d93be83406a84d8bf" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 41931 } } peers { permanent_uuid: "3640643bc9624931a62e720a75b7a3d9" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 40265 } }
I20260928 06:35:59.402583 31420 ts_tablet_manager.cc:1428] T d76506ad98434625ad6941ca0970bd03 P 599bfef32939446d93be83406a84d8bf: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
Sep 28 06:35:59 dist-test-slave-rrj5 krb5kdc[30760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577359, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.65@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:35:59.413452 31533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:59.397308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:48877 (local address 127.16.157.126:43889)
0928 06:35:59.401943 (+ 4635us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:59.401954 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:59.402002 (+ 48us) server_negotiation.cc:407] Connection header received
0928 06:35:59.402167 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:59.402175 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:59.402240 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:59.402401 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:59.403491 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:59.404875 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:59.406664 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:59.407219 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:59.407334 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:59.409955 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:59.409959 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:59.409959 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:59.409970 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:59.411671 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:59.412115 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:59.412118 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:59.412119 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:59.412205 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:59.412476 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:59.412477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:59.412477 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:59.412699 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:59.412790 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:59.413126 (+ 336us) server_negotiation.cc:299] Negotiation successful
0928 06:35:59.413268 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4571,"thread_start_us":98,"threads_started":1}
I20260928 06:35:59.413467 31531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:59.395378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:43889 (local address 127.16.157.65:48877)
0928 06:35:59.401109 (+ 5731us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:59.401161 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:59.401346 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:59.402569 (+ 1223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:59.402574 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:59.403015 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:59.403343 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:59.403350 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:59.405004 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:59.405006 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:59.406537 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:59.406541 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:59.407331 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:59.407332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:59.407491 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:59.408171 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:59.408183 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:59.409834 (+ 1651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:59.411776 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:59.411783 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:59.411785 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:59.412014 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:59.412290 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:59.412291 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:59.412291 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:59.412402 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:59.412783 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:59.412790 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:59.412948 (+ 158us) client_negotiation.cc:769] Sending connection context
0928 06:35:59.413054 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 06:35:59.413236 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5674,"thread_start_us":87,"threads_started":1}
I20260928 06:35:59.414445 31529 heartbeater.cc:347] Connected to a master server at 127.16.157.126:43889
I20260928 06:35:59.414534 31529 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:59.414732 31529 heartbeater.cc:511] Master 127.16.157.126:43889 requested a full tablet report, sending...
I20260928 06:35:59.415406 31340 ts_manager.cc:194] Registered new tserver with Master: 599bfef32939446d93be83406a84d8bf (127.16.157.65:41931)
I20260928 06:35:59.416463 31340 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.157.65@KRBTEST.COM'} at 127.16.157.65:48877
I20260928 06:35:59.417409 31529 heartbeater.cc:503] Master 127.16.157.126:43889 was elected leader, sending a full tablet report...
Sep 28 06:35:59 dist-test-slave-rrj5 krb5kdc[30760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577359, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.126@KRBTEST.COM for kudu/127.16.157.65@KRBTEST.COM
I20260928 06:35:59.424392 31535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:59.416025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.65:41931 (local address 127.0.0.1:48886)
0928 06:35:59.416234 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:59.416273 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:59.416334 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:59.416527 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:59.416532 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:59.416870 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:59.416917 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:59.416919 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:59.418271 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:59.418272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:59.418830 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:59.418832 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:59.419474 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:59.419475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:59.419528 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:59.420016 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:59.420020 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:59.421377 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:59.423284 (+ 1907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:59.423291 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:59.423293 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:59.423528 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:59.423765 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:59.423765 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:59.423766 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:59.423823 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:59.424121 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:59.424128 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:59.424169 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 06:35:59.424233 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 06:35:59.424324 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":119,"threads_started":1}
I20260928 06:35:59.424482 31536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:59.416101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48886 (local address 127.16.157.65:41931)
0928 06:35:59.416294 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:59.416299 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:59.416314 (+ 15us) server_negotiation.cc:407] Connection header received
0928 06:35:59.416423 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:59.416425 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:59.416438 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:59.416510 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:59.417006 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:59.418190 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:59.418916 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:59.419394 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:59.419468 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:59.421458 (+ 1990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:59.421465 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:59.421465 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:59.421472 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:59.423193 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:59.423610 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:59.423612 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:59.423613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:59.423679 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:59.423884 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:59.423885 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:59.423885 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:59.424058 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:59.424147 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:59.424281 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 06:35:59.424381 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":164,"thread_start_us":82,"threads_started":1}
I20260928 06:35:59.424865 31462 tablet_service.cc:1514] Processing DeleteTablet for tablet d76506ad98434625ad6941ca0970bd03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:59 UTC) from {username='kudu', principal='kudu/127.16.157.126@KRBTEST.COM'} at 127.0.0.1:48886
I20260928 06:35:59.425524 31537 tablet_replica.cc:330] stopping tablet replica
I20260928 06:35:59.425675 31537 raft_consensus.cc:2229] T d76506ad98434625ad6941ca0970bd03 P 599bfef32939446d93be83406a84d8bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:35:59.425750 31537 raft_consensus.cc:2258] T d76506ad98434625ad6941ca0970bd03 P 599bfef32939446d93be83406a84d8bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:59.426111 31537 ts_tablet_manager.cc:1905] T d76506ad98434625ad6941ca0970bd03 P 599bfef32939446d93be83406a84d8bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:59.428172 31537 ts_tablet_manager.cc:1918] T d76506ad98434625ad6941ca0970bd03 P 599bfef32939446d93be83406a84d8bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:35:59.428284 31537 log.cc:1196] T d76506ad98434625ad6941ca0970bd03 P 599bfef32939446d93be83406a84d8bf: Deleting WAL directory at /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-0/wal/wals/d76506ad98434625ad6941ca0970bd03
I20260928 06:35:59.428524 31537 ts_tablet_manager.cc:1939] T d76506ad98434625ad6941ca0970bd03 P 599bfef32939446d93be83406a84d8bf: Deleting consensus metadata
I20260928 06:35:59.429013 31325 catalog_manager.cc:4841] TS 599bfef32939446d93be83406a84d8bf (127.16.157.65:41931): tablet d76506ad98434625ad6941ca0970bd03 (table test-table [id=8db8dfaddcfc4222ab9e361d3b868dd7]) successfully deleted
W20260928 06:35:59.486039 31532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:59.486301 31532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:59.486346 31532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:59.487494 31532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:59.487565 31532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:59.487623 31532 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.66
I20260928 06:35:59.489102 31532 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:37571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab
--principal=kudu/127.16.157.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.66:40265
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.157.66
--webserver_port=35161
--tserver_master_addrs=127.16.157.126:43889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.66
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:35:59.489368 31532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:59.489647 31532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:59.494738 31542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:59.494746 31543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:59.494917 31532 server_base.cc:1034] running on GCE node
W20260928 06:35:59.494854 31545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:59.495232 31532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:59.495487 31532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:59.496665 31532 hybrid_clock.cc:648] HybridClock initialized: now 1790577359496645 us; error 38 us; skew 500 ppm
Sep 28 06:35:59 dist-test-slave-rrj5 krb5kdc[30760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577359, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:59.499289 31532 init.cc:377] Logged in from keytab as kudu/127.16.157.66@KRBTEST.COM (short username kudu)
I20260928 06:35:59.499729 31532 webserver.cc:460] Webserver started at http://127.16.157.66:35161/ using document root <none> and password file <none>
I20260928 06:35:59.500010 31532 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:59.500090 31532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:59.501451 31532 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:35:59.502125 31552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:59.502305 31532 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:35:59.502372 31532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-1/wal
uuid: "3640643bc9624931a62e720a75b7a3d9"
format_stamp: "Formatted at 2026-09-28 06:35:58 on dist-test-slave-rrj5"
server_key: "7fb3e8e258f201951cc66ca655d142df"
server_key_iv: "266ec53330d2f74db71d2954d53d0873"
server_key_version: "encryptionkey@0"
I20260928 06:35:59.502736 31532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:59.514478 31532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:59.516366 31532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:59.516520 31532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:59.516778 31532 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:59.517094 31532 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:59.517127 31532 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:59.517168 31532 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:59.517195 31532 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:59.523295 31532 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.66:40265
I20260928 06:35:59.523345 31665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.66:40265 every 8 connection(s)
I20260928 06:35:59.523681 31532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
Sep 28 06:35:59 dist-test-slave-rrj5 krb5kdc[30760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577359, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.66@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:35:59.532778 31533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:59.524445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.66:33385 (local address 127.16.157.126:43889)
0928 06:35:59.524496 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:59.524498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:59.524696 (+ 198us) server_negotiation.cc:407] Connection header received
0928 06:35:59.525214 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:59.525214 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:59.525226 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:59.525284 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:59.526278 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:59.526806 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:59.527764 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:59.527951 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:59.528004 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:59.530108 (+ 2104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:59.530109 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:59.530109 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:59.530116 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:59.531540 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:59.531882 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:59.531883 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:59.531883 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:59.531931 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:59.532135 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:59.532136 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:59.532136 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:59.532230 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:59.532277 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:59.532634 (+ 357us) server_negotiation.cc:299] Negotiation successful
0928 06:35:59.532713 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:35:59.532786 31668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:59.524402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:43889 (local address 127.16.157.66:33385)
0928 06:35:59.524577 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:59.524620 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:59.525126 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:59.525343 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:59.525346 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:59.525986 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:59.526180 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:59.526186 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:59.526890 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:59.526891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:59.527687 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:59.527688 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:59.528027 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:59.528027 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:59.528126 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:59.528587 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:59.528595 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:59.530014 (+ 1419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:59.531610 (+ 1596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:59.531616 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:59.531617 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:59.531820 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:59.531989 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:59.531989 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:59.531989 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:59.532082 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:59.532292 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:59.532296 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:59.532405 (+ 109us) client_negotiation.cc:769] Sending connection context
0928 06:35:59.532486 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 06:35:59.532622 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":97,"threads_started":1}
I20260928 06:35:59.533090 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 31532
I20260928 06:35:59.533396 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.67:42925
--local_ip_for_outbound_sockets=127.16.157.67
--tserver_master_addrs=127.16.157.126:43889
--webserver_port=39901
--webserver_interface=127.16.157.67
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab
--principal=kudu/127.16.157.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:37571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:35:59.534065 31666 heartbeater.cc:347] Connected to a master server at 127.16.157.126:43889
I20260928 06:35:59.534184 31666 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:59.534384 31666 heartbeater.cc:511] Master 127.16.157.126:43889 requested a full tablet report, sending...
I20260928 06:35:59.534816 31340 ts_manager.cc:194] Registered new tserver with Master: 3640643bc9624931a62e720a75b7a3d9 (127.16.157.66:40265)
I20260928 06:35:59.535269 31340 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.157.66@KRBTEST.COM'} at 127.16.157.66:33385
W20260928 06:35:59.616480 31669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:59.616689 31669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:59.616714 31669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:59.617897 31669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:59.617939 31669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:59.617966 31669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.67
I20260928 06:35:59.619493 31669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:37571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab
--principal=kudu/127.16.157.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.67:42925
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.157.67
--webserver_port=39901
--tserver_master_addrs=127.16.157.126:43889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.67
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:35:59.619741 31669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:59.620003 31669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:59.625205 31677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:59.625223 31674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:59.625254 31669 server_base.cc:1034] running on GCE node
W20260928 06:35:59.625262 31675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:59.625675 31669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:59.625934 31669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:59.627082 31669 hybrid_clock.cc:648] HybridClock initialized: now 1790577359627065 us; error 33 us; skew 500 ppm
Sep 28 06:35:59 dist-test-slave-rrj5 krb5kdc[30760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577359, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:59.629696 31669 init.cc:377] Logged in from keytab as kudu/127.16.157.67@KRBTEST.COM (short username kudu)
I20260928 06:35:59.630123 31669 webserver.cc:460] Webserver started at http://127.16.157.67:39901/ using document root <none> and password file <none>
I20260928 06:35:59.630354 31669 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:59.630427 31669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:59.631809 31669 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:35:59.632408 31684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:59.632578 31669 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:35:59.632700 31669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-2/wal
uuid: "cea63d9ad0874e73af2ef12dbda8feca"
format_stamp: "Formatted at 2026-09-28 06:35:58 on dist-test-slave-rrj5"
server_key: "f6c58ed13093b7fef3f24b32345450fb"
server_key_iv: "4f604774cf6c3ff4d5eb0178f41e3ef5"
server_key_version: "encryptionkey@0"
I20260928 06:35:59.633033 31669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:59.655488 31669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:59.657404 31669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:59.657536 31669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:59.657823 31669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:59.658154 31669 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:59.658212 31669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:59.658255 31669 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:59.658284 31669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:59.664377 31669 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.67:42925
I20260928 06:35:59.664427 31797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.67:42925 every 8 connection(s)
I20260928 06:35:59.664779 31669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
I20260928 06:35:59.669219 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 31669
Sep 28 06:35:59 dist-test-slave-rrj5 krb5kdc[30760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577359, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.67@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:35:59.675320 31533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:59.665464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:35811 (local address 127.16.157.126:43889)
0928 06:35:59.665508 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:59.665510 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:59.665780 (+ 270us) server_negotiation.cc:407] Connection header received
0928 06:35:59.666378 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:59.666380 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:59.666393 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:59.666444 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:59.667528 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:59.668058 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:59.669091 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:59.669296 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:59.669373 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:59.671941 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:59.671942 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:59.671942 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:59.671950 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:59.673471 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:59.674084 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:59.674086 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:59.674086 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:59.674145 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:59.674530 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:59.674531 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:59.674531 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:59.674729 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:59.674833 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:59.675103 (+ 270us) server_negotiation.cc:299] Negotiation successful
0928 06:35:59.675198 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:35:59.675349 31800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:59.665450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:43889 (local address 127.16.157.67:35811)
0928 06:35:59.665639 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:59.665686 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:59.666243 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:59.666534 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:59.666539 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:59.667229 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:59.667425 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:59.667430 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:59.668155 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:59.668156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:59.668984 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:59.668986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:59.669416 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:59.669418 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:59.669600 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:59.670218 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:59.670227 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:59.671846 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:59.673599 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:59.673609 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:59.673611 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:59.673992 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:59.674221 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:59.674222 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:59.674222 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:59.674433 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:59.674810 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:59.674816 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:59.674943 (+ 127us) client_negotiation.cc:769] Sending connection context
0928 06:35:59.675033 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 06:35:59.675160 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":75,"threads_started":1}
I20260928 06:35:59.675880 31798 heartbeater.cc:347] Connected to a master server at 127.16.157.126:43889
I20260928 06:35:59.676015 31798 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:59.676205 31798 heartbeater.cc:511] Master 127.16.157.126:43889 requested a full tablet report, sending...
I20260928 06:35:59.676577 31802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:59.669668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47802 (local address 127.16.157.126:43889)
0928 06:35:59.669819 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:59.669821 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:59.669859 (+ 38us) server_negotiation.cc:407] Connection header received
0928 06:35:59.669928 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:59.669929 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:59.669938 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:59.670069 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:59.670630 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:59.671178 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:59.672171 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:59.672369 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:59.672441 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:59.673534 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:59.673535 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:59.673535 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:59.673542 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:59.675357 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:59.675811 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:59.675812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:59.675813 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:59.675869 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:59.676045 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:59.676046 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:59.676046 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:59.676168 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:59.676351 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:59.676388 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 06:35:59.676466 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":102,"threads_started":1}
I20260928 06:35:59.676621 31340 ts_manager.cc:194] Registered new tserver with Master: cea63d9ad0874e73af2ef12dbda8feca (127.16.157.67:42925)
I20260928 06:35:59.677106 31340 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.157.67@KRBTEST.COM'} at 127.16.157.67:35811
I20260928 06:35:59.677280 17013 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:35:59.679872 31802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:59.677645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47818 (local address 127.16.157.126:43889)
0928 06:35:59.677685 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:59.677687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:59.677854 (+ 167us) server_negotiation.cc:407] Connection header received
0928 06:35:59.677891 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:59.677892 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:59.677927 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:59.678009 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:35:59.678078 (+ 69us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:59.678578 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:59.679262 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:59.679450 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:59.679501 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:59.679631 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:35:59.679721 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:35:59.679785 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:59.679806 (+ 21us) server_negotiation.cc:299] Negotiation successful
0928 06:35:59.679826 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:35:59.680217 31340 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:47818:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:35:59.681964 31340 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:35:59.684267 31462 tablet_service.cc:1467] Processing CreateTablet for tablet fa6f03a5a3d04a2bb22204c897cc0d91 (DEFAULT_TABLE table=test-table [id=aec10b264be844768d811ef064f8d762]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:59.684494 31462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa6f03a5a3d04a2bb22204c897cc0d91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:59.686331 31420 tablet_bootstrap.cc:492] T fa6f03a5a3d04a2bb22204c897cc0d91 P 599bfef32939446d93be83406a84d8bf: Bootstrap starting.
I20260928 06:35:59.686786 31420 tablet_bootstrap.cc:654] T fa6f03a5a3d04a2bb22204c897cc0d91 P 599bfef32939446d93be83406a84d8bf: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:59.687860 31806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:59.684098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48766 (local address 127.16.157.67:42925)
0928 06:35:59.684277 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:59.684281 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:59.684291 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:35:59.684339 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:59.684341 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:59.684353 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:59.684477 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:59.684606 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:59.685492 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:59.686730 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:59.687453 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:59.687563 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:59.687634 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:59.687757 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 06:35:59.687796 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":108,"threads_started":1}
I20260928 06:35:59.687858 31535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:59.683869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.67:42925 (local address 127.0.0.1:48766)
0928 06:35:59.683928 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:59.683940 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:59.683992 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:59.684486 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:59.684487 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:59.684489 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:59.684548 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:59.684550 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:59.685619 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:59.685620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:59.686633 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:59.686637 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:59.687569 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:59.687570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:59.687644 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:59.687668 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 06:35:59.687745 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 06:35:59.687788 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 06:35:59.688347 31420 tablet_bootstrap.cc:492] T fa6f03a5a3d04a2bb22204c897cc0d91 P 599bfef32939446d93be83406a84d8bf: No bootstrap required, opened a new log
I20260928 06:35:59.688407 31420 ts_tablet_manager.cc:1397] T fa6f03a5a3d04a2bb22204c897cc0d91 P 599bfef32939446d93be83406a84d8bf: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 06:35:59.688544 31420 raft_consensus.cc:348] T fa6f03a5a3d04a2bb22204c897cc0d91 P 599bfef32939446d93be83406a84d8bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea63d9ad0874e73af2ef12dbda8feca" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 42925 } } peers { permanent_uuid: "3640643bc9624931a62e720a75b7a3d9" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 40265 } } peers { permanent_uuid: "599bfef32939446d93be83406a84d8bf" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 41931 } }
I20260928 06:35:59.688596 31420 raft_consensus.cc:374] T fa6f03a5a3d04a2bb22204c897cc0d91 P 599bfef32939446d93be83406a84d8bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:59.688627 31420 raft_consensus.cc:729] T fa6f03a5a3d04a2bb22204c897cc0d91 P 599bfef32939446d93be83406a84d8bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 599bfef32939446d93be83406a84d8bf, State: Initialized, Role: FOLLOWER
I20260928 06:35:59.688699 31420 consensus_queue.cc:260] T fa6f03a5a3d04a2bb22204c897cc0d91 P 599bfef32939446d93be83406a84d8bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea63d9ad0874e73af2ef12dbda8feca" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 42925 } } peers { permanent_uuid: "3640643bc9624931a62e720a75b7a3d9" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 40265 } } peers { permanent_uuid: "599bfef32939446d93be83406a84d8bf" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 41931 } }
I20260928 06:35:59.688629 31732 tablet_service.cc:1467] Processing CreateTablet for tablet fa6f03a5a3d04a2bb22204c897cc0d91 (DEFAULT_TABLE table=test-table [id=aec10b264be844768d811ef064f8d762]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:59.688796 31420 ts_tablet_manager.cc:1428] T fa6f03a5a3d04a2bb22204c897cc0d91 P 599bfef32939446d93be83406a84d8bf: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:59.688997 31732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa6f03a5a3d04a2bb22204c897cc0d91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:59.689597 31804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:59.683869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.66:40265 (local address 127.0.0.1:42446)
0928 06:35:59.684091 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:59.684100 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:59.684141 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:59.685895 (+ 1754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:59.685896 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:59.685898 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:59.685935 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:59.685936 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:59.687214 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:59.687215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:59.688305 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:59.688308 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:59.689326 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:59.689327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:59.689398 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:59.689402 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:35:59.689456 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 06:35:59.689497 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":25,"thread_start_us":94,"threads_started":1}
I20260928 06:35:59.689836 31805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:59.684089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42446 (local address 127.16.157.66:40265)
0928 06:35:59.685711 (+ 1622us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:59.685716 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:59.685729 (+ 13us) server_negotiation.cc:407] Connection header received
0928 06:35:59.685768 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:59.685779 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:59.685795 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:59.685882 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:59.686000 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:59.687074 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:59.688414 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:59.689145 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:59.689219 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:59.689285 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:59.689708 (+ 423us) server_negotiation.cc:299] Negotiation successful
0928 06:35:59.689758 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1580,"thread_start_us":98,"threads_started":1}
I20260928 06:35:59.692032 31808 tablet_bootstrap.cc:492] T fa6f03a5a3d04a2bb22204c897cc0d91 P cea63d9ad0874e73af2ef12dbda8feca: Bootstrap starting.
I20260928 06:35:59.692624 31600 tablet_service.cc:1467] Processing CreateTablet for tablet fa6f03a5a3d04a2bb22204c897cc0d91 (DEFAULT_TABLE table=test-table [id=aec10b264be844768d811ef064f8d762]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:59.692763 31808 tablet_bootstrap.cc:654] T fa6f03a5a3d04a2bb22204c897cc0d91 P cea63d9ad0874e73af2ef12dbda8feca: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:59.692955 31600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa6f03a5a3d04a2bb22204c897cc0d91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:59.693253 31808 log.cc:824] T fa6f03a5a3d04a2bb22204c897cc0d91 P cea63d9ad0874e73af2ef12dbda8feca: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:59.694069 31808 tablet_bootstrap.cc:492] T fa6f03a5a3d04a2bb22204c897cc0d91 P cea63d9ad0874e73af2ef12dbda8feca: No bootstrap required, opened a new log
I20260928 06:35:59.694124 31808 ts_tablet_manager.cc:1397] T fa6f03a5a3d04a2bb22204c897cc0d91 P cea63d9ad0874e73af2ef12dbda8feca: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:35:59.695425 31808 raft_consensus.cc:348] T fa6f03a5a3d04a2bb22204c897cc0d91 P cea63d9ad0874e73af2ef12dbda8feca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea63d9ad0874e73af2ef12dbda8feca" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 42925 } } peers { permanent_uuid: "3640643bc9624931a62e720a75b7a3d9" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 40265 } } peers { permanent_uuid: "599bfef32939446d93be83406a84d8bf" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 41931 } }
I20260928 06:35:59.695551 31808 raft_consensus.cc:374] T fa6f03a5a3d04a2bb22204c897cc0d91 P cea63d9ad0874e73af2ef12dbda8feca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:59.695572 31808 raft_consensus.cc:729] T fa6f03a5a3d04a2bb22204c897cc0d91 P cea63d9ad0874e73af2ef12dbda8feca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea63d9ad0874e73af2ef12dbda8feca, State: Initialized, Role: FOLLOWER
I20260928 06:35:59.695782 31808 consensus_queue.cc:260] T fa6f03a5a3d04a2bb22204c897cc0d91 P cea63d9ad0874e73af2ef12dbda8feca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea63d9ad0874e73af2ef12dbda8feca" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 42925 } } peers { permanent_uuid: "3640643bc9624931a62e720a75b7a3d9" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 40265 } } peers { permanent_uuid: "599bfef32939446d93be83406a84d8bf" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 41931 } }
I20260928 06:35:59.695852 31810 tablet_bootstrap.cc:492] T fa6f03a5a3d04a2bb22204c897cc0d91 P 3640643bc9624931a62e720a75b7a3d9: Bootstrap starting.
I20260928 06:35:59.696101 31808 ts_tablet_manager.cc:1428] T fa6f03a5a3d04a2bb22204c897cc0d91 P cea63d9ad0874e73af2ef12dbda8feca: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:35:59.696242 31798 heartbeater.cc:503] Master 127.16.157.126:43889 was elected leader, sending a full tablet report...
I20260928 06:35:59.696563 31810 tablet_bootstrap.cc:654] T fa6f03a5a3d04a2bb22204c897cc0d91 P 3640643bc9624931a62e720a75b7a3d9: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:59.697183 31810 log.cc:824] T fa6f03a5a3d04a2bb22204c897cc0d91 P 3640643bc9624931a62e720a75b7a3d9: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:59.697935 31810 tablet_bootstrap.cc:492] T fa6f03a5a3d04a2bb22204c897cc0d91 P 3640643bc9624931a62e720a75b7a3d9: No bootstrap required, opened a new log
I20260928 06:35:59.698029 31810 ts_tablet_manager.cc:1397] T fa6f03a5a3d04a2bb22204c897cc0d91 P 3640643bc9624931a62e720a75b7a3d9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:35:59.699384 31810 raft_consensus.cc:348] T fa6f03a5a3d04a2bb22204c897cc0d91 P 3640643bc9624931a62e720a75b7a3d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea63d9ad0874e73af2ef12dbda8feca" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 42925 } } peers { permanent_uuid: "3640643bc9624931a62e720a75b7a3d9" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 40265 } } peers { permanent_uuid: "599bfef32939446d93be83406a84d8bf" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 41931 } }
I20260928 06:35:59.699546 31810 raft_consensus.cc:374] T fa6f03a5a3d04a2bb22204c897cc0d91 P 3640643bc9624931a62e720a75b7a3d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:59.699595 31810 raft_consensus.cc:729] T fa6f03a5a3d04a2bb22204c897cc0d91 P 3640643bc9624931a62e720a75b7a3d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3640643bc9624931a62e720a75b7a3d9, State: Initialized, Role: FOLLOWER
I20260928 06:35:59.699723 31810 consensus_queue.cc:260] T fa6f03a5a3d04a2bb22204c897cc0d91 P 3640643bc9624931a62e720a75b7a3d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea63d9ad0874e73af2ef12dbda8feca" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 42925 } } peers { permanent_uuid: "3640643bc9624931a62e720a75b7a3d9" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 40265 } } peers { permanent_uuid: "599bfef32939446d93be83406a84d8bf" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 41931 } }
I20260928 06:35:59.700048 31810 ts_tablet_manager.cc:1428] T fa6f03a5a3d04a2bb22204c897cc0d91 P 3640643bc9624931a62e720a75b7a3d9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:35:59.700115 31666 heartbeater.cc:503] Master 127.16.157.126:43889 was elected leader, sending a full tablet report...
W20260928 06:35:59.774294 31667 tablet.cc:2369] T fa6f03a5a3d04a2bb22204c897cc0d91 P 3640643bc9624931a62e720a75b7a3d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:35:59.812477 31811 raft_consensus.cc:482] T fa6f03a5a3d04a2bb22204c897cc0d91 P cea63d9ad0874e73af2ef12dbda8feca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:35:59.812673 31811 raft_consensus.cc:504] T fa6f03a5a3d04a2bb22204c897cc0d91 P cea63d9ad0874e73af2ef12dbda8feca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea63d9ad0874e73af2ef12dbda8feca" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 42925 } } peers { permanent_uuid: "3640643bc9624931a62e720a75b7a3d9" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 40265 } } peers { permanent_uuid: "599bfef32939446d93be83406a84d8bf" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 41931 } }
I20260928 06:35:59.813160 31811 leader_election.cc:288] T fa6f03a5a3d04a2bb22204c897cc0d91 P cea63d9ad0874e73af2ef12dbda8feca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3640643bc9624931a62e720a75b7a3d9 (127.16.157.66:40265), 599bfef32939446d93be83406a84d8bf (127.16.157.65:41931)
I20260928 06:35:59.816973 31805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:59.813445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:44851 (local address 127.16.157.66:40265)
0928 06:35:59.813488 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:59.813490 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:59.813501 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:35:59.813621 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:59.813622 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:59.813632 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:59.813686 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:59.813879 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:59.814632 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:59.816020 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:59.816589 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:59.816671 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:59.816680 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:59.816869 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 06:35:59.816913 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:35:59.816989 31536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:59.813471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:33985 (local address 127.16.157.65:41931)
0928 06:35:59.813532 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:59.813535 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:59.813758 (+ 223us) server_negotiation.cc:407] Connection header received
0928 06:35:59.813796 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:59.813797 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:59.813819 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:59.813904 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:59.814063 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:59.814752 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:59.816017 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:59.816663 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:59.816729 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:59.816799 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:59.816853 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 06:35:59.816901 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 06:35:59.816982 31800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:59.813372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.66:40265 (local address 127.16.157.67:44851)
0928 06:35:59.813433 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:59.813445 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:59.813491 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:59.813710 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:59.813712 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:59.813714 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:59.813794 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:59.813796 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:59.814716 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:59.814717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:59.815899 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:59.815902 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:59.816672 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:59.816672 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:59.816777 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:59.816781 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:35:59.816840 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 06:35:59.816887 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":8}
I20260928 06:35:59.816996 31814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:59.813404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.65:41931 (local address 127.16.157.67:33985)
0928 06:35:59.813629 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:59.813639 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:59.813734 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:59.813920 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:59.813920 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:59.813922 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:59.813970 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:59.813971 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:59.814830 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:59.814831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:59.815899 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:59.815902 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:59.816736 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:59.816737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:59.816790 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:59.816793 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:35:59.816840 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 06:35:59.816885 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":5,"thread_start_us":128,"threads_started":1}
I20260928 06:35:59.817325 31620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa6f03a5a3d04a2bb22204c897cc0d91" candidate_uuid: "cea63d9ad0874e73af2ef12dbda8feca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3640643bc9624931a62e720a75b7a3d9" is_pre_election: true
I20260928 06:35:59.817492 31620 raft_consensus.cc:2454] T fa6f03a5a3d04a2bb22204c897cc0d91 P 3640643bc9624931a62e720a75b7a3d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea63d9ad0874e73af2ef12dbda8feca in term 0.
I20260928 06:35:59.817613 31482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa6f03a5a3d04a2bb22204c897cc0d91" candidate_uuid: "cea63d9ad0874e73af2ef12dbda8feca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "599bfef32939446d93be83406a84d8bf" is_pre_election: true
I20260928 06:35:59.817775 31685 leader_election.cc:302] T fa6f03a5a3d04a2bb22204c897cc0d91 P cea63d9ad0874e73af2ef12dbda8feca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3640643bc9624931a62e720a75b7a3d9, cea63d9ad0874e73af2ef12dbda8feca; no voters:
I20260928 06:35:59.817801 31482 raft_consensus.cc:2454] T fa6f03a5a3d04a2bb22204c897cc0d91 P 599bfef32939446d93be83406a84d8bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea63d9ad0874e73af2ef12dbda8feca in term 0.
I20260928 06:35:59.817937 31811 raft_consensus.cc:2781] T fa6f03a5a3d04a2bb22204c897cc0d91 P cea63d9ad0874e73af2ef12dbda8feca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:35:59.818032 31811 raft_consensus.cc:482] T fa6f03a5a3d04a2bb22204c897cc0d91 P cea63d9ad0874e73af2ef12dbda8feca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:35:59.818065 31811 raft_consensus.cc:3037] T fa6f03a5a3d04a2bb22204c897cc0d91 P cea63d9ad0874e73af2ef12dbda8feca [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:59.818795 31811 raft_consensus.cc:504] T fa6f03a5a3d04a2bb22204c897cc0d91 P cea63d9ad0874e73af2ef12dbda8feca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea63d9ad0874e73af2ef12dbda8feca" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 42925 } } peers { permanent_uuid: "3640643bc9624931a62e720a75b7a3d9" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 40265 } } peers { permanent_uuid: "599bfef32939446d93be83406a84d8bf" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 41931 } }
I20260928 06:35:59.818950 31811 leader_election.cc:288] T fa6f03a5a3d04a2bb22204c897cc0d91 P cea63d9ad0874e73af2ef12dbda8feca [CANDIDATE]: Term 1 election: Requested vote from peers 3640643bc9624931a62e720a75b7a3d9 (127.16.157.66:40265), 599bfef32939446d93be83406a84d8bf (127.16.157.65:41931)
I20260928 06:35:59.819113 31620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa6f03a5a3d04a2bb22204c897cc0d91" candidate_uuid: "cea63d9ad0874e73af2ef12dbda8feca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3640643bc9624931a62e720a75b7a3d9"
I20260928 06:35:59.819113 31482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa6f03a5a3d04a2bb22204c897cc0d91" candidate_uuid: "cea63d9ad0874e73af2ef12dbda8feca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "599bfef32939446d93be83406a84d8bf"
I20260928 06:35:59.819188 31482 raft_consensus.cc:3037] T fa6f03a5a3d04a2bb22204c897cc0d91 P 599bfef32939446d93be83406a84d8bf [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:59.819190 31620 raft_consensus.cc:3037] T fa6f03a5a3d04a2bb22204c897cc0d91 P 3640643bc9624931a62e720a75b7a3d9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:59.820091 31482 raft_consensus.cc:2454] T fa6f03a5a3d04a2bb22204c897cc0d91 P 599bfef32939446d93be83406a84d8bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea63d9ad0874e73af2ef12dbda8feca in term 1.
I20260928 06:35:59.820091 31620 raft_consensus.cc:2454] T fa6f03a5a3d04a2bb22204c897cc0d91 P 3640643bc9624931a62e720a75b7a3d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea63d9ad0874e73af2ef12dbda8feca in term 1.
I20260928 06:35:59.820283 31685 leader_election.cc:302] T fa6f03a5a3d04a2bb22204c897cc0d91 P cea63d9ad0874e73af2ef12dbda8feca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3640643bc9624931a62e720a75b7a3d9, cea63d9ad0874e73af2ef12dbda8feca; no voters:
I20260928 06:35:59.820372 31811 raft_consensus.cc:2781] T fa6f03a5a3d04a2bb22204c897cc0d91 P cea63d9ad0874e73af2ef12dbda8feca [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:59.820515 31811 raft_consensus.cc:686] T fa6f03a5a3d04a2bb22204c897cc0d91 P cea63d9ad0874e73af2ef12dbda8feca [term 1 LEADER]: Becoming Leader. State: Replica: cea63d9ad0874e73af2ef12dbda8feca, State: Running, Role: LEADER
I20260928 06:35:59.820614 31811 consensus_queue.cc:237] T fa6f03a5a3d04a2bb22204c897cc0d91 P cea63d9ad0874e73af2ef12dbda8feca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea63d9ad0874e73af2ef12dbda8feca" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 42925 } } peers { permanent_uuid: "3640643bc9624931a62e720a75b7a3d9" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 40265 } } peers { permanent_uuid: "599bfef32939446d93be83406a84d8bf" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 41931 } }
I20260928 06:35:59.821537 31339 catalog_manager.cc:5495] T fa6f03a5a3d04a2bb22204c897cc0d91 P cea63d9ad0874e73af2ef12dbda8feca reported cstate change: term changed from 0 to 1, leader changed from <none> to cea63d9ad0874e73af2ef12dbda8feca (127.16.157.67). New cstate: current_term: 1 leader_uuid: "cea63d9ad0874e73af2ef12dbda8feca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea63d9ad0874e73af2ef12dbda8feca" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 42925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3640643bc9624931a62e720a75b7a3d9" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 40265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "599bfef32939446d93be83406a84d8bf" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 41931 } health_report { overall_health: UNKNOWN } } }
I20260928 06:35:59.833264 31806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:59.831162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48768 (local address 127.16.157.67:42925)
0928 06:35:59.831222 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:59.831224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:59.831231 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:35:59.831263 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:59.831263 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:59.831287 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:59.831348 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:35:59.831457 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:59.831919 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:59.832577 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:59.832756 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:59.832806 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:59.832984 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:35:59.833080 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:35:59.833139 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:59.833171 (+ 32us) server_negotiation.cc:299] Negotiation successful
0928 06:35:59.833188 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 06:35:59.835029 31620 raft_consensus.cc:1261] T fa6f03a5a3d04a2bb22204c897cc0d91 P 3640643bc9624931a62e720a75b7a3d9 [term 1 FOLLOWER]: Refusing update from remote peer cea63d9ad0874e73af2ef12dbda8feca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:35:59.835064 31482 raft_consensus.cc:1261] T fa6f03a5a3d04a2bb22204c897cc0d91 P 599bfef32939446d93be83406a84d8bf [term 1 FOLLOWER]: Refusing update from remote peer cea63d9ad0874e73af2ef12dbda8feca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:35:59.835399 31811 consensus_queue.cc:1035] T fa6f03a5a3d04a2bb22204c897cc0d91 P cea63d9ad0874e73af2ef12dbda8feca [LEADER]: Connected to new peer: Peer: permanent_uuid: "3640643bc9624931a62e720a75b7a3d9" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 40265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:35:59.835554 31815 consensus_queue.cc:1035] T fa6f03a5a3d04a2bb22204c897cc0d91 P cea63d9ad0874e73af2ef12dbda8feca [LEADER]: Connected to new peer: Peer: permanent_uuid: "599bfef32939446d93be83406a84d8bf" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:35:59.838514 31820 mvcc.cc:205] Tried to move back new op lower bound from 7334204865881829376 to 7334204865825869824. Current Snapshot: MvccSnapshot[applied={T|T < 7334204865881829376}]
I20260928 06:35:59.839869 31819 mvcc.cc:205] Tried to move back new op lower bound from 7334204865881829376 to 7334204865825869824. Current Snapshot: MvccSnapshot[applied={T|T < 7334204865881829376 or (T in {7334204865881829376})}]
I20260928 06:35:59.840063 31339 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:47818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:35:59.840149 31339 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:47818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:35:59.840843 31339 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ab2320c32a4546c7add6995d595eaec1: Sending DeleteTablet for 3 replicas of tablet fa6f03a5a3d04a2bb22204c897cc0d91
I20260928 06:35:59.841147 31462 tablet_service.cc:1514] Processing DeleteTablet for tablet fa6f03a5a3d04a2bb22204c897cc0d91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:59 UTC) from {username='kudu', principal='kudu/127.16.157.126@KRBTEST.COM'} at 127.0.0.1:48886
I20260928 06:35:59.841163 31600 tablet_service.cc:1514] Processing DeleteTablet for tablet fa6f03a5a3d04a2bb22204c897cc0d91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:59 UTC) from {username='kudu', principal='kudu/127.16.157.126@KRBTEST.COM'} at 127.0.0.1:42446
I20260928 06:35:59.841221 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 31378
I20260928 06:35:59.841364 31537 tablet_replica.cc:330] T fa6f03a5a3d04a2bb22204c897cc0d91 P 599bfef32939446d93be83406a84d8bf: stopping tablet replica
I20260928 06:35:59.841427 31537 raft_consensus.cc:2229] T fa6f03a5a3d04a2bb22204c897cc0d91 P 599bfef32939446d93be83406a84d8bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:35:59.841472 31537 raft_consensus.cc:2258] T fa6f03a5a3d04a2bb22204c897cc0d91 P 599bfef32939446d93be83406a84d8bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:59.841881 31732 tablet_service.cc:1514] Processing DeleteTablet for tablet fa6f03a5a3d04a2bb22204c897cc0d91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:59 UTC) from {username='kudu', principal='kudu/127.16.157.126@KRBTEST.COM'} at 127.0.0.1:48766
I20260928 06:35:59.842080 31828 tablet_replica.cc:330] T fa6f03a5a3d04a2bb22204c897cc0d91 P 3640643bc9624931a62e720a75b7a3d9: stopping tablet replica
I20260928 06:35:59.842209 31828 raft_consensus.cc:2229] T fa6f03a5a3d04a2bb22204c897cc0d91 P 3640643bc9624931a62e720a75b7a3d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:35:59.842350 31828 raft_consensus.cc:2258] T fa6f03a5a3d04a2bb22204c897cc0d91 P 3640643bc9624931a62e720a75b7a3d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:59.842561 31829 tablet_replica.cc:330] T fa6f03a5a3d04a2bb22204c897cc0d91 P cea63d9ad0874e73af2ef12dbda8feca: stopping tablet replica
I20260928 06:35:59.842687 31829 raft_consensus.cc:2229] T fa6f03a5a3d04a2bb22204c897cc0d91 P cea63d9ad0874e73af2ef12dbda8feca [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:35:59.842808 31829 raft_consensus.cc:2258] T fa6f03a5a3d04a2bb22204c897cc0d91 P cea63d9ad0874e73af2ef12dbda8feca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:59.843197 31829 ts_tablet_manager.cc:1905] T fa6f03a5a3d04a2bb22204c897cc0d91 P cea63d9ad0874e73af2ef12dbda8feca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:59.845287 31829 ts_tablet_manager.cc:1918] T fa6f03a5a3d04a2bb22204c897cc0d91 P cea63d9ad0874e73af2ef12dbda8feca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:35:59.845403 31829 log.cc:1196] T fa6f03a5a3d04a2bb22204c897cc0d91 P cea63d9ad0874e73af2ef12dbda8feca: Deleting WAL directory at /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-2/wal/wals/fa6f03a5a3d04a2bb22204c897cc0d91
I20260928 06:35:59.845754 31829 ts_tablet_manager.cc:1939] T fa6f03a5a3d04a2bb22204c897cc0d91 P cea63d9ad0874e73af2ef12dbda8feca: Deleting consensus metadata
I20260928 06:35:59.845978 31828 ts_tablet_manager.cc:1905] T fa6f03a5a3d04a2bb22204c897cc0d91 P 3640643bc9624931a62e720a75b7a3d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:59.846259 31325 catalog_manager.cc:4841] TS cea63d9ad0874e73af2ef12dbda8feca (127.16.157.67:42925): tablet fa6f03a5a3d04a2bb22204c897cc0d91 (table test-table [id=aec10b264be844768d811ef064f8d762]) successfully deleted
I20260928 06:35:59.847640 31828 ts_tablet_manager.cc:1918] T fa6f03a5a3d04a2bb22204c897cc0d91 P 3640643bc9624931a62e720a75b7a3d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:35:59.847720 31828 log.cc:1196] T fa6f03a5a3d04a2bb22204c897cc0d91 P 3640643bc9624931a62e720a75b7a3d9: Deleting WAL directory at /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-1/wal/wals/fa6f03a5a3d04a2bb22204c897cc0d91
I20260928 06:35:59.848055 31828 ts_tablet_manager.cc:1939] T fa6f03a5a3d04a2bb22204c897cc0d91 P 3640643bc9624931a62e720a75b7a3d9: Deleting consensus metadata
I20260928 06:35:59.848551 31324 catalog_manager.cc:4841] TS 3640643bc9624931a62e720a75b7a3d9 (127.16.157.66:40265): tablet fa6f03a5a3d04a2bb22204c897cc0d91 (table test-table [id=aec10b264be844768d811ef064f8d762]) successfully deleted
W20260928 06:35:59.848632 31325 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.157.65:41931 (error 108)
I20260928 06:35:59.849181 31804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:59.849037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.65:41931 (local address 127.0.0.1:48888)
0928 06:35:59.849103 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:59.849141 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.65:41931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 06:35:59.849706 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 31532
W20260928 06:35:59.849326 31325 catalog_manager.cc:4584] TS 599bfef32939446d93be83406a84d8bf (127.16.157.65:41931): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fa6f03a5a3d04a2bb22204c897cc0d91: Network error: Client connection negotiation failed: client connection to 127.16.157.65:41931: connect: Connection refused (error 111)
I20260928 06:35:59.855939 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 31669
I20260928 06:35:59.861140 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 31307
I20260928 06:35:59.866812 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:43889
--webserver_interface=127.16.157.126
--webserver_port=42253
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=kudu/127.16.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:37571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.157.126:43889
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:59.948000 31831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:59.948216 31831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:59.948239 31831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:59.949411 31831 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:59.949455 31831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:59.949468 31831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:59.949523 31831 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:59.949550 31831 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:35:59.951032 31831 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:37571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.157.126:43889
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=kudu/127.16.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:43889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.157.126
--webserver_port=42253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:35:59.951304 31831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:59.951560 31831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:59.956385 31839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:59.956432 31837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:59.956416 31836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:59.956634 31831 server_base.cc:1034] running on GCE node
I20260928 06:35:59.956878 31831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:59.957161 31831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:59.958307 31831 hybrid_clock.cc:648] HybridClock initialized: now 1790577359958301 us; error 37 us; skew 500 ppm
Sep 28 06:35:59 dist-test-slave-rrj5 krb5kdc[30760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577359, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:59.960829 31831 init.cc:377] Logged in from keytab as kudu/127.16.157.126@KRBTEST.COM (short username kudu)
I20260928 06:35:59.961308 31831 webserver.cc:460] Webserver started at http://127.16.157.126:42253/ using document root <none> and password file <none>
I20260928 06:35:59.961556 31831 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:59.961690 31831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:59.963042 31831 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:35:59.963661 31846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:59.963864 31831 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:35:59.963935 31831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/master-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/master-0/wal
uuid: "ab2320c32a4546c7add6995d595eaec1"
format_stamp: "Formatted at 2026-09-28 06:35:58 on dist-test-slave-rrj5"
server_key: "9ff8de3d86ed1367a1bfc03942f0e033"
server_key_iv: "ffe5bd95634356bb1a41af4f7722e39c"
server_key_version: "encryptionkey@0"
I20260928 06:35:59.964377 31831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:59.978746 31831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:59.980481 31831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:59.980593 31831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:59.984761 31831 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.126:43889
I20260928 06:35:59.984803 31898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.126:43889 every 8 connection(s)
I20260928 06:35:59.985190 31831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
I20260928 06:35:59.986877 31899 sys_catalog.cc:263] Verifying existing consensus state
I20260928 06:35:59.987483 31899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab2320c32a4546c7add6995d595eaec1: Bootstrap starting.
I20260928 06:35:59.992419 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 31831
I20260928 06:35:59.992733 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.65:41931
--local_ip_for_outbound_sockets=127.16.157.65
--tserver_master_addrs=127.16.157.126:43889
--webserver_port=44931
--webserver_interface=127.16.157.65
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab
--principal=kudu/127.16.157.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:37571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:35:59.996922 31899 log.cc:824] T 00000000000000000000000000000000 P ab2320c32a4546c7add6995d595eaec1: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:59.999411 31899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab2320c32a4546c7add6995d595eaec1: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:35:59.999732 31899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab2320c32a4546c7add6995d595eaec1: Bootstrap complete.
I20260928 06:36:00.002162 31899 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab2320c32a4546c7add6995d595eaec1 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab2320c32a4546c7add6995d595eaec1" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 43889 } }
I20260928 06:36:00.002422 31899 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab2320c32a4546c7add6995d595eaec1 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab2320c32a4546c7add6995d595eaec1, State: Initialized, Role: FOLLOWER
I20260928 06:36:00.002627 31899 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab2320c32a4546c7add6995d595eaec1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab2320c32a4546c7add6995d595eaec1" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 43889 } }
I20260928 06:36:00.002727 31899 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab2320c32a4546c7add6995d595eaec1 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:36:00.002782 31899 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab2320c32a4546c7add6995d595eaec1 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:36:00.002832 31899 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab2320c32a4546c7add6995d595eaec1 [term 2 FOLLOWER]: Advancing to term 3
I20260928 06:36:00.003841 31899 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab2320c32a4546c7add6995d595eaec1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab2320c32a4546c7add6995d595eaec1" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 43889 } }
I20260928 06:36:00.003974 31899 leader_election.cc:302] T 00000000000000000000000000000000 P ab2320c32a4546c7add6995d595eaec1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab2320c32a4546c7add6995d595eaec1; no voters:
I20260928 06:36:00.004189 31899 leader_election.cc:288] T 00000000000000000000000000000000 P ab2320c32a4546c7add6995d595eaec1 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 06:36:00.004230 31903 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab2320c32a4546c7add6995d595eaec1 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 06:36:00.004348 31903 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab2320c32a4546c7add6995d595eaec1 [term 3 LEADER]: Becoming Leader. State: Replica: ab2320c32a4546c7add6995d595eaec1, State: Running, Role: LEADER
I20260928 06:36:00.004443 31903 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab2320c32a4546c7add6995d595eaec1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab2320c32a4546c7add6995d595eaec1" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 43889 } }
I20260928 06:36:00.004489 31899 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab2320c32a4546c7add6995d595eaec1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:36:00.004700 31904 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab2320c32a4546c7add6995d595eaec1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab2320c32a4546c7add6995d595eaec1. Latest consensus state: current_term: 3 leader_uuid: "ab2320c32a4546c7add6995d595eaec1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab2320c32a4546c7add6995d595eaec1" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 43889 } } }
I20260928 06:36:00.004908 31904 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab2320c32a4546c7add6995d595eaec1 [sys.catalog]: This master's current role is: LEADER
I20260928 06:36:00.004853 31905 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab2320c32a4546c7add6995d595eaec1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "ab2320c32a4546c7add6995d595eaec1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab2320c32a4546c7add6995d595eaec1" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 43889 } } }
I20260928 06:36:00.005157 31905 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab2320c32a4546c7add6995d595eaec1 [sys.catalog]: This master's current role is: LEADER
I20260928 06:36:00.005308 31911 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:36:00.006163 31911 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:36:00.006261 31911 catalog_manager.cc:1259] Loaded cluster ID: 407bc57bc9364616a89fa3a8874ffe05
I20260928 06:36:00.006304 31911 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 06:36:00.006677 31911 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ffa248a69ad google::LogMessage::Fail() at ??:0
@ 0x7ffa248a988a google::LogMessage::SendToLog() at ??:0
@ 0x7ffa248a64de google::LogMessage::Flush() at ??:0
@ 0x7ffa248a7a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ffa2c7c70f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7ffa2c7d08b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7ffa267740d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ffa26769954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7ffa275cd6db start_thread at ??:0
@ 0x7ffa233a461f clone at ??:0
W20260928 06:36:00.090932 31901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:36:00.091150 31901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:36:00.091187 31901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:36:00.092342 31901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:36:00.092384 31901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:36:00.092410 31901 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.65
I20260928 06:36:00.093974 31901 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:37571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab
--principal=kudu/127.16.157.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.65:41931
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.157.65
--webserver_port=44931
--tserver_master_addrs=127.16.157.126:43889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.65
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:36:00.094234 31901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:36:00.094486 31901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:36:00.099753 31936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:00.099764 31935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:00.099937 31938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:00.100364 31901 server_base.cc:1034] running on GCE node
I20260928 06:36:00.100607 31901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:36:00.100881 31901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:36:00.102033 31901 hybrid_clock.cc:648] HybridClock initialized: now 1790577360102020 us; error 38 us; skew 500 ppm
Sep 28 06:36:00 dist-test-slave-rrj5 krb5kdc[30760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577360, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:36:00.104547 31901 init.cc:377] Logged in from keytab as kudu/127.16.157.65@KRBTEST.COM (short username kudu)
I20260928 06:36:00.105001 31901 webserver.cc:460] Webserver started at http://127.16.157.65:44931/ using document root <none> and password file <none>
I20260928 06:36:00.105211 31901 fs_manager.cc:359] Metadata directory not provided
I20260928 06:36:00.105252 31901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:36:00.106621 31901 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:36:00.107215 31945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:00.107437 31901 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:36:00.107527 31901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-0/wal
uuid: "599bfef32939446d93be83406a84d8bf"
format_stamp: "Formatted at 2026-09-28 06:35:58 on dist-test-slave-rrj5"
server_key: "6721edab5f4d3f725bce9616aeb78a79"
server_key_iv: "33e8eca60a1d89a4cce1d8288d2d6e29"
server_key_version: "encryptionkey@0"
I20260928 06:36:00.107858 31901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:36:00.130170 31901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:36:00.132705 31901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:36:00.132874 31901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:36:00.133169 31901 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:36:00.133668 31952 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 06:36:00.135720 31901 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 06:36:00.135790 31901 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 06:36:00.135838 31901 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 06:36:00.136512 31901 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 06:36:00.136572 31901 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 06:36:00.136650 31952 tablet_bootstrap.cc:492] T fa6f03a5a3d04a2bb22204c897cc0d91 P 599bfef32939446d93be83406a84d8bf: Bootstrap starting.
I20260928 06:36:00.144075 31901 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.65:41931
I20260928 06:36:00.144176 32059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.65:41931 every 8 connection(s)
I20260928 06:36:00.144534 31901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
I20260928 06:36:00.148360 31952 log.cc:824] T fa6f03a5a3d04a2bb22204c897cc0d91 P 599bfef32939446d93be83406a84d8bf: Log is configured to *not* fsync() on all Append() calls
I20260928 06:36:00.148654 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 31901
I20260928 06:36:00.148994 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.66:40265
--local_ip_for_outbound_sockets=127.16.157.66
--tserver_master_addrs=127.16.157.126:43889
--webserver_port=35161
--webserver_interface=127.16.157.66
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab
--principal=kudu/127.16.157.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:37571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:36:00.151212 31952 tablet_bootstrap.cc:492] T fa6f03a5a3d04a2bb22204c897cc0d91 P 599bfef32939446d93be83406a84d8bf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:36:00.151523 31952 tablet_bootstrap.cc:492] T fa6f03a5a3d04a2bb22204c897cc0d91 P 599bfef32939446d93be83406a84d8bf: Bootstrap complete.
I20260928 06:36:00.152524 31952 ts_tablet_manager.cc:1397] T fa6f03a5a3d04a2bb22204c897cc0d91 P 599bfef32939446d93be83406a84d8bf: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.013s
I20260928 06:36:00.153964 31952 raft_consensus.cc:348] T fa6f03a5a3d04a2bb22204c897cc0d91 P 599bfef32939446d93be83406a84d8bf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea63d9ad0874e73af2ef12dbda8feca" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 42925 } } peers { permanent_uuid: "3640643bc9624931a62e720a75b7a3d9" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 40265 } } peers { permanent_uuid: "599bfef32939446d93be83406a84d8bf" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 41931 } }
I20260928 06:36:00.154253 31952 raft_consensus.cc:729] T fa6f03a5a3d04a2bb22204c897cc0d91 P 599bfef32939446d93be83406a84d8bf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 599bfef32939446d93be83406a84d8bf, State: Initialized, Role: FOLLOWER
I20260928 06:36:00.154413 31952 consensus_queue.cc:260] T fa6f03a5a3d04a2bb22204c897cc0d91 P 599bfef32939446d93be83406a84d8bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea63d9ad0874e73af2ef12dbda8feca" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 42925 } } peers { permanent_uuid: "3640643bc9624931a62e720a75b7a3d9" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 40265 } } peers { permanent_uuid: "599bfef32939446d93be83406a84d8bf" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 41931 } }
I20260928 06:36:00.154728 31952 ts_tablet_manager.cc:1428] T fa6f03a5a3d04a2bb22204c897cc0d91 P 599bfef32939446d93be83406a84d8bf: Time spent starting tablet: real 0.002s user 0.002s sys 0.003s
W20260928 06:36:00.240760 32062 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:00.145338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:43889 (local address 127.16.157.65:60349)
0928 06:36:00.145512 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:00.145602 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:00.146379 (+ 777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:00.240559 (+ 94180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:43889: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":80,"threads_started":1}
W20260928 06:36:00.241035 32060 heartbeater.cc:644] Failed to heartbeat to 127.16.157.126:43889 (0 consecutive failures): Network error: Failed to ping master at 127.16.157.126:43889: Client connection negotiation failed: client connection to 127.16.157.126:43889: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 06:36:00.241407 32062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:00.241289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:43889 (local address 127.16.157.65:41305)
0928 06:36:00.241343 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:00.241388 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:43889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 06:36:00.241814 32062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:00.241728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:43889 (local address 127.16.157.65:50849)
0928 06:36:00.241777 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:00.241800 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:43889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 06:36:00.241946 32060 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 06:36:00.249543 32064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:36:00.249812 32064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:36:00.249835 32064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:36:00.250991 32064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:36:00.251031 32064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:36:00.251085 32064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.66
I20260928 06:36:00.252596 32064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:37571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab
--principal=kudu/127.16.157.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.66:40265
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.157.66
--webserver_port=35161
--tserver_master_addrs=127.16.157.126:43889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.66
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:36:00.252835 32064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:36:00.253060 32064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:36:00.258134 32070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:00.258159 32073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:00.258291 32071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:00.258397 32064 server_base.cc:1034] running on GCE node
I20260928 06:36:00.258664 32064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:36:00.258917 32064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:36:00.260080 32064 hybrid_clock.cc:648] HybridClock initialized: now 1790577360260067 us; error 47 us; skew 500 ppm
Sep 28 06:36:00 dist-test-slave-rrj5 krb5kdc[30760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577360, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:36:00.262681 32064 init.cc:377] Logged in from keytab as kudu/127.16.157.66@KRBTEST.COM (short username kudu)
I20260928 06:36:00.263119 32064 webserver.cc:460] Webserver started at http://127.16.157.66:35161/ using document root <none> and password file <none>
I20260928 06:36:00.263366 32064 fs_manager.cc:359] Metadata directory not provided
I20260928 06:36:00.263440 32064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:36:00.264834 32064 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:36:00.265465 32080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:00.265641 32064 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:36:00.265750 32064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-1/wal
uuid: "3640643bc9624931a62e720a75b7a3d9"
format_stamp: "Formatted at 2026-09-28 06:35:58 on dist-test-slave-rrj5"
server_key: "7fb3e8e258f201951cc66ca655d142df"
server_key_iv: "266ec53330d2f74db71d2954d53d0873"
server_key_version: "encryptionkey@0"
I20260928 06:36:00.266106 32064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:36:00.279302 32064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:36:00.281910 32064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:36:00.282087 32064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:36:00.282331 32064 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:36:00.282648 32064 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:36:00.282709 32064 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:00.282755 32064 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:36:00.282778 32064 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:00.289175 32064 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.66:40265
I20260928 06:36:00.289255 32193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.66:40265 every 8 connection(s)
I20260928 06:36:00.289707 32064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
I20260928 06:36:00.290823 32196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:00.290420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:43889 (local address 127.16.157.66:37719)
0928 06:36:00.290579 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:00.290659 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:43889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":75,"threads_started":1}
W20260928 06:36:00.291240 32194 heartbeater.cc:644] Failed to heartbeat to 127.16.157.126:43889 (0 consecutive failures): Network error: Failed to ping master at 127.16.157.126:43889: Client connection negotiation failed: client connection to 127.16.157.126:43889: connect: Connection refused (error 111)
I20260928 06:36:00.291527 32196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:00.291444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:43889 (local address 127.16.157.66:32933)
0928 06:36:00.291486 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:00.291511 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:43889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 06:36:00.291860 32196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:00.291770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:43889 (local address 127.16.157.66:54223)
0928 06:36:00.291806 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:00.291837 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:43889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260928 06:36:00.291987 32194 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 06:36:00.295485 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 32064
I20260928 06:36:00.295707 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.67:42925
--local_ip_for_outbound_sockets=127.16.157.67
--tserver_master_addrs=127.16.157.126:43889
--webserver_port=39901
--webserver_interface=127.16.157.67
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab
--principal=kudu/127.16.157.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:37571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:36:00.380328 32197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:36:00.380528 32197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:36:00.380550 32197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:36:00.381789 32197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:36:00.381843 32197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:36:00.381888 32197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.67
I20260928 06:36:00.383353 32197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:37571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab
--principal=kudu/127.16.157.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.67:42925
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.157.67
--webserver_port=39901
--tserver_master_addrs=127.16.157.126:43889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.67
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:36:00.383615 32197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:36:00.383854 32197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:36:00.389182 32203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:00.389205 32205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:00.389340 32197 server_base.cc:1034] running on GCE node
W20260928 06:36:00.389394 32202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:00.389737 32197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:36:00.390019 32197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:36:00.391196 32197 hybrid_clock.cc:648] HybridClock initialized: now 1790577360391175 us; error 42 us; skew 500 ppm
Sep 28 06:36:00 dist-test-slave-rrj5 krb5kdc[30760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577360, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:36:00.393846 32197 init.cc:377] Logged in from keytab as kudu/127.16.157.67@KRBTEST.COM (short username kudu)
I20260928 06:36:00.394371 32197 webserver.cc:460] Webserver started at http://127.16.157.67:39901/ using document root <none> and password file <none>
I20260928 06:36:00.394629 32197 fs_manager.cc:359] Metadata directory not provided
I20260928 06:36:00.394707 32197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:36:00.396148 32197 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:36:00.396806 32212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:00.396983 32197 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:36:00.397091 32197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-2/wal
uuid: "cea63d9ad0874e73af2ef12dbda8feca"
format_stamp: "Formatted at 2026-09-28 06:35:58 on dist-test-slave-rrj5"
server_key: "f6c58ed13093b7fef3f24b32345450fb"
server_key_iv: "4f604774cf6c3ff4d5eb0178f41e3ef5"
server_key_version: "encryptionkey@0"
I20260928 06:36:00.397452 32197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:36:00.410089 32197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:36:00.412173 32197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:36:00.412330 32197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:36:00.412585 32197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:36:00.412902 32197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:36:00.412961 32197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:00.413002 32197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:36:00.413029 32197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:00.419320 32197 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.67:42925
I20260928 06:36:00.419375 32325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.67:42925 every 8 connection(s)
I20260928 06:36:00.419781 32197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
I20260928 06:36:00.420872 32328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:00.420496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:43889 (local address 127.16.157.67:44759)
0928 06:36:00.420656 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:00.420734 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:43889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":77,"threads_started":1}
W20260928 06:36:00.421298 32326 heartbeater.cc:644] Failed to heartbeat to 127.16.157.126:43889 (0 consecutive failures): Network error: Failed to ping master at 127.16.157.126:43889: Client connection negotiation failed: client connection to 127.16.157.126:43889: connect: Connection refused (error 111)
I20260928 06:36:00.421618 32328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:00.421480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:43889 (local address 127.16.157.67:52961)
0928 06:36:00.421531 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:00.421557 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:43889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 06:36:00.421914 32328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:00.421837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:43889 (local address 127.16.157.67:58381)
0928 06:36:00.421879 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:00.421901 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:43889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 06:36:00.422158 32326 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 06:36:00.425166 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 32197
I20260928 06:36:01.242484 32330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:01.242271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:43889 (local address 127.16.157.65:36203)
0928 06:36:01.242402 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:01.242438 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:43889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":61,"threads_started":1}
I20260928 06:36:01.292477 32331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:01.292276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:43889 (local address 127.16.157.66:48085)
0928 06:36:01.292401 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:01.292436 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:43889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":74,"threads_started":1}
I20260928 06:36:01.422641 32332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:01.422437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:43889 (local address 127.16.157.67:53191)
0928 06:36:01.422563 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:01.422595 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:43889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":69,"threads_started":1}
I20260928 06:36:01.759399 32333 raft_consensus.cc:482] T fa6f03a5a3d04a2bb22204c897cc0d91 P 599bfef32939446d93be83406a84d8bf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:36:01.759547 32333 raft_consensus.cc:504] T fa6f03a5a3d04a2bb22204c897cc0d91 P 599bfef32939446d93be83406a84d8bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea63d9ad0874e73af2ef12dbda8feca" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 42925 } } peers { permanent_uuid: "3640643bc9624931a62e720a75b7a3d9" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 40265 } } peers { permanent_uuid: "599bfef32939446d93be83406a84d8bf" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 41931 } }
I20260928 06:36:01.759960 32333 leader_election.cc:288] T fa6f03a5a3d04a2bb22204c897cc0d91 P 599bfef32939446d93be83406a84d8bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea63d9ad0874e73af2ef12dbda8feca (127.16.157.67:42925), 3640643bc9624931a62e720a75b7a3d9 (127.16.157.66:40265)
Sep 28 06:36:01 dist-test-slave-rrj5 krb5kdc[30760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577360, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.65@KRBTEST.COM for kudu/127.16.157.66@KRBTEST.COM
Sep 28 06:36:01 dist-test-slave-rrj5 krb5kdc[30760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577360, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.65@KRBTEST.COM for kudu/127.16.157.67@KRBTEST.COM
I20260928 06:36:01.771184 32336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:01.760142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.66:40265 (local address 127.16.157.65:42323)
0928 06:36:01.760304 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:01.760313 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:01.760390 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:01.761263 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:01.761264 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:01.761687 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:01.761903 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:01.761908 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:01.763195 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:01.763196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:01.764053 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:01.764055 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:01.764659 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:01.764660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:01.764794 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:01.765418 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:01.765422 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:01.767022 (+ 1600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:36:01.769360 (+ 2338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:01.769368 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:01.769370 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:01.769746 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:01.770014 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:01.770015 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:01.770015 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:01.770151 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:01.770615 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:36:01.770621 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:36:01.770758 (+ 137us) client_negotiation.cc:769] Sending connection context
0928 06:36:01.770862 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 06:36:01.771075 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":36,"thread_start_us":78,"threads_started":1}
I20260928 06:36:01.771282 32337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:01.760265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:42323 (local address 127.16.157.66:40265)
0928 06:36:01.760438 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:01.760449 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:01.760487 (+ 38us) server_negotiation.cc:407] Connection header received
0928 06:36:01.760916 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:01.760918 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:01.760960 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:01.761115 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:01.762040 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:01.763088 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:01.764144 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:01.764566 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:01.764686 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:01.767151 (+ 2465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:01.767156 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:01.767156 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:01.767169 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:01.769267 (+ 2098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:01.769818 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:01.769821 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:01.769822 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:01.769939 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:01.770250 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:01.770251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:01.770252 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:01.770542 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:01.770609 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:01.770997 (+ 388us) server_negotiation.cc:299] Negotiation successful
0928 06:36:01.771160 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":74,"threads_started":1}
I20260928 06:36:01.771440 32335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:01.760123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:60985 (local address 127.16.157.67:42925)
0928 06:36:01.760332 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:01.760346 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:01.760391 (+ 45us) server_negotiation.cc:407] Connection header received
0928 06:36:01.760919 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:01.760922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:01.760962 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:01.761151 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:01.762078 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:01.763092 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:01.764178 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:01.764594 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:01.764727 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:01.767614 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:01.767618 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:01.767618 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:01.767629 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:01.769530 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:01.770025 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:01.770029 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:01.770029 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:01.770132 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:01.770419 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:01.770420 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:01.770420 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:01.770727 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:01.770823 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:01.771156 (+ 333us) server_negotiation.cc:299] Negotiation successful
0928 06:36:01.771293 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":55,"threads_started":1}
I20260928 06:36:01.771309 32334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:01.759990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.67:42925 (local address 127.16.157.65:60985)
0928 06:36:01.760131 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:01.760143 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:01.760205 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:01.761181 (+ 976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:01.761185 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:01.761686 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:01.761902 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:01.761909 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:01.763222 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:01.763223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:01.764054 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:01.764056 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:01.764700 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:01.764701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:01.764797 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:01.765357 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:01.765365 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:01.767479 (+ 2114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:36:01.769653 (+ 2174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:01.769654 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:01.769654 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:01.769932 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:01.770228 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:01.770229 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:01.770230 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:01.770309 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:01.770903 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:36:01.770904 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:36:01.770996 (+ 92us) client_negotiation.cc:769] Sending connection context
0928 06:36:01.771082 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 06:36:01.771176 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":43,"thread_start_us":54,"threads_started":1}
W20260928 06:36:01.772727 31946 leader_election.cc:341] T fa6f03a5a3d04a2bb22204c897cc0d91 P 599bfef32939446d93be83406a84d8bf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3640643bc9624931a62e720a75b7a3d9 (127.16.157.66:40265): Not found: Tablet not found: fa6f03a5a3d04a2bb22204c897cc0d91
W20260928 06:36:01.772820 31947 leader_election.cc:341] T fa6f03a5a3d04a2bb22204c897cc0d91 P 599bfef32939446d93be83406a84d8bf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cea63d9ad0874e73af2ef12dbda8feca (127.16.157.67:42925): Not found: Tablet not found: fa6f03a5a3d04a2bb22204c897cc0d91
I20260928 06:36:01.772873 31947 leader_election.cc:302] T fa6f03a5a3d04a2bb22204c897cc0d91 P 599bfef32939446d93be83406a84d8bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 599bfef32939446d93be83406a84d8bf; no voters: 3640643bc9624931a62e720a75b7a3d9, cea63d9ad0874e73af2ef12dbda8feca
I20260928 06:36:01.772984 32333 raft_consensus.cc:2726] T fa6f03a5a3d04a2bb22204c897cc0d91 P 599bfef32939446d93be83406a84d8bf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:36:02.243108 32334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:02.242960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:43889 (local address 127.16.157.65:43223)
0928 06:36:02.243039 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:02.243070 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:43889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260928 06:36:02.293080 32338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:02.292888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:43889 (local address 127.16.157.66:34249)
0928 06:36:02.293009 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:02.293046 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:43889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":55,"threads_started":1}
I20260928 06:36:02.423338 32339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:02.423141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:43889 (local address 127.16.157.67:41161)
0928 06:36:02.423269 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:02.423304 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:43889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":60,"threads_started":1}
I20260928 06:36:03.243831 32340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:03.243605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:43889 (local address 127.16.157.65:55481)
0928 06:36:03.243746 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:03.243784 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:43889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":63,"threads_started":1}
I20260928 06:36:03.293707 32341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:03.293497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:43889 (local address 127.16.157.66:50923)
0928 06:36:03.293639 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:03.293671 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:43889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":54,"threads_started":1}
I20260928 06:36:03.424048 32342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:03.423847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:43889 (local address 127.16.157.67:44809)
0928 06:36:03.423975 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:03.424012 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:43889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":61,"threads_started":1}
I20260928 06:36:03.547276 32343 raft_consensus.cc:482] T fa6f03a5a3d04a2bb22204c897cc0d91 P 599bfef32939446d93be83406a84d8bf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:36:03.547370 32343 raft_consensus.cc:504] T fa6f03a5a3d04a2bb22204c897cc0d91 P 599bfef32939446d93be83406a84d8bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea63d9ad0874e73af2ef12dbda8feca" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 42925 } } peers { permanent_uuid: "3640643bc9624931a62e720a75b7a3d9" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 40265 } } peers { permanent_uuid: "599bfef32939446d93be83406a84d8bf" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 41931 } }
I20260928 06:36:03.547505 32343 leader_election.cc:288] T fa6f03a5a3d04a2bb22204c897cc0d91 P 599bfef32939446d93be83406a84d8bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea63d9ad0874e73af2ef12dbda8feca (127.16.157.67:42925), 3640643bc9624931a62e720a75b7a3d9 (127.16.157.66:40265)
W20260928 06:36:03.547951 31947 leader_election.cc:341] T fa6f03a5a3d04a2bb22204c897cc0d91 P 599bfef32939446d93be83406a84d8bf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cea63d9ad0874e73af2ef12dbda8feca (127.16.157.67:42925): Not found: Tablet not found: fa6f03a5a3d04a2bb22204c897cc0d91
W20260928 06:36:03.548076 31946 leader_election.cc:341] T fa6f03a5a3d04a2bb22204c897cc0d91 P 599bfef32939446d93be83406a84d8bf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3640643bc9624931a62e720a75b7a3d9 (127.16.157.66:40265): Not found: Tablet not found: fa6f03a5a3d04a2bb22204c897cc0d91
I20260928 06:36:03.548148 31946 leader_election.cc:302] T fa6f03a5a3d04a2bb22204c897cc0d91 P 599bfef32939446d93be83406a84d8bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 599bfef32939446d93be83406a84d8bf; no voters: 3640643bc9624931a62e720a75b7a3d9, cea63d9ad0874e73af2ef12dbda8feca
I20260928 06:36:03.548216 32343 raft_consensus.cc:2726] T fa6f03a5a3d04a2bb22204c897cc0d91 P 599bfef32939446d93be83406a84d8bf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:36:04.244469 32344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:04.244278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:43889 (local address 127.16.157.65:52591)
0928 06:36:04.244394 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:04.244423 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:43889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":56,"threads_started":1}
I20260928 06:36:04.294361 32345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:04.294111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:43889 (local address 127.16.157.66:57879)
0928 06:36:04.294283 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:04.294322 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:43889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":71,"threads_started":1}
I20260928 06:36:04.424716 32346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:04.424503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:43889 (local address 127.16.157.67:37787)
0928 06:36:04.424643 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:04.424677 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:43889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":72,"threads_started":1}
I20260928 06:36:05.245174 32347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:05.244972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:43889 (local address 127.16.157.65:50531)
0928 06:36:05.245093 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:05.245128 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:43889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":67,"threads_started":1}
I20260928 06:36:05.294971 32348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:05.294770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:43889 (local address 127.16.157.66:34121)
0928 06:36:05.294902 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:05.294936 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:43889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":74,"threads_started":1}
I20260928 06:36:05.425313 32349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:05.425120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:43889 (local address 127.16.157.67:46425)
0928 06:36:05.425243 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:05.425275 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:43889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":78,"threads_started":1}
I20260928 06:36:05.729642 32350 raft_consensus.cc:482] T fa6f03a5a3d04a2bb22204c897cc0d91 P 599bfef32939446d93be83406a84d8bf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:36:05.729739 32350 raft_consensus.cc:504] T fa6f03a5a3d04a2bb22204c897cc0d91 P 599bfef32939446d93be83406a84d8bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea63d9ad0874e73af2ef12dbda8feca" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 42925 } } peers { permanent_uuid: "3640643bc9624931a62e720a75b7a3d9" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 40265 } } peers { permanent_uuid: "599bfef32939446d93be83406a84d8bf" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 41931 } }
I20260928 06:36:05.729871 32350 leader_election.cc:288] T fa6f03a5a3d04a2bb22204c897cc0d91 P 599bfef32939446d93be83406a84d8bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea63d9ad0874e73af2ef12dbda8feca (127.16.157.67:42925), 3640643bc9624931a62e720a75b7a3d9 (127.16.157.66:40265)
W20260928 06:36:05.730365 31946 leader_election.cc:341] T fa6f03a5a3d04a2bb22204c897cc0d91 P 599bfef32939446d93be83406a84d8bf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3640643bc9624931a62e720a75b7a3d9 (127.16.157.66:40265): Not found: Tablet not found: fa6f03a5a3d04a2bb22204c897cc0d91
W20260928 06:36:05.730463 31947 leader_election.cc:341] T fa6f03a5a3d04a2bb22204c897cc0d91 P 599bfef32939446d93be83406a84d8bf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cea63d9ad0874e73af2ef12dbda8feca (127.16.157.67:42925): Not found: Tablet not found: fa6f03a5a3d04a2bb22204c897cc0d91
I20260928 06:36:05.730535 31947 leader_election.cc:302] T fa6f03a5a3d04a2bb22204c897cc0d91 P 599bfef32939446d93be83406a84d8bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 599bfef32939446d93be83406a84d8bf; no voters: 3640643bc9624931a62e720a75b7a3d9, cea63d9ad0874e73af2ef12dbda8feca
I20260928 06:36:05.730619 32350 raft_consensus.cc:2726] T fa6f03a5a3d04a2bb22204c897cc0d91 P 599bfef32939446d93be83406a84d8bf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:36:06.245934 32351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:06.245711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:43889 (local address 127.16.157.65:58311)
0928 06:36:06.245853 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:06.245888 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:43889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":80,"threads_started":1}
I20260928 06:36:06.295555 32352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:06.295360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:43889 (local address 127.16.157.66:46145)
0928 06:36:06.295485 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:06.295518 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:43889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":75,"threads_started":1}
I20260928 06:36:06.425899 32353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:06.425682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:43889 (local address 127.16.157.67:39309)
0928 06:36:06.425828 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:06.425861 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:43889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":73,"threads_started":1}
I20260928 06:36:07.246596 32354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:07.246377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:43889 (local address 127.16.157.65:34993)
0928 06:36:07.246503 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:07.246533 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:43889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":73,"threads_started":1}
I20260928 06:36:07.296387 32355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:07.296130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:43889 (local address 127.16.157.66:33269)
0928 06:36:07.296281 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:07.296323 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:43889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":86,"threads_started":1}
I20260928 06:36:07.426503 32356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:07.426282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:43889 (local address 127.16.157.67:54305)
0928 06:36:07.426427 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:07.426466 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:43889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":71,"threads_started":1}
I20260928 06:36:08.247386 32357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:08.247112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:43889 (local address 127.16.157.65:52027)
0928 06:36:08.247295 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:08.247337 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:43889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":84,"threads_started":1}
I20260928 06:36:08.297050 32358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:08.296832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:43889 (local address 127.16.157.66:60153)
0928 06:36:08.296971 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:08.297007 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:43889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":66,"threads_started":1}
I20260928 06:36:08.427145 32359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:08.426921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:43889 (local address 127.16.157.67:58353)
0928 06:36:08.427050 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:08.427103 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:43889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":63,"threads_started":1}
I20260928 06:36:09.248266 32360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:09.247917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:43889 (local address 127.16.157.65:49337)
0928 06:36:09.248167 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:09.248215 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:43889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":61,"threads_started":1}
I20260928 06:36:09.288136 32361 raft_consensus.cc:482] T fa6f03a5a3d04a2bb22204c897cc0d91 P 599bfef32939446d93be83406a84d8bf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:36:09.288237 32361 raft_consensus.cc:504] T fa6f03a5a3d04a2bb22204c897cc0d91 P 599bfef32939446d93be83406a84d8bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea63d9ad0874e73af2ef12dbda8feca" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 42925 } } peers { permanent_uuid: "3640643bc9624931a62e720a75b7a3d9" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 40265 } } peers { permanent_uuid: "599bfef32939446d93be83406a84d8bf" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 41931 } }
I20260928 06:36:09.288447 32361 leader_election.cc:288] T fa6f03a5a3d04a2bb22204c897cc0d91 P 599bfef32939446d93be83406a84d8bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea63d9ad0874e73af2ef12dbda8feca (127.16.157.67:42925), 3640643bc9624931a62e720a75b7a3d9 (127.16.157.66:40265)
W20260928 06:36:09.288904 31947 leader_election.cc:341] T fa6f03a5a3d04a2bb22204c897cc0d91 P 599bfef32939446d93be83406a84d8bf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cea63d9ad0874e73af2ef12dbda8feca (127.16.157.67:42925): Not found: Tablet not found: fa6f03a5a3d04a2bb22204c897cc0d91
W20260928 06:36:09.288982 31946 leader_election.cc:341] T fa6f03a5a3d04a2bb22204c897cc0d91 P 599bfef32939446d93be83406a84d8bf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3640643bc9624931a62e720a75b7a3d9 (127.16.157.66:40265): Not found: Tablet not found: fa6f03a5a3d04a2bb22204c897cc0d91
I20260928 06:36:09.289028 31946 leader_election.cc:302] T fa6f03a5a3d04a2bb22204c897cc0d91 P 599bfef32939446d93be83406a84d8bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 599bfef32939446d93be83406a84d8bf; no voters: 3640643bc9624931a62e720a75b7a3d9, cea63d9ad0874e73af2ef12dbda8feca
I20260928 06:36:09.289121 32361 raft_consensus.cc:2726] T fa6f03a5a3d04a2bb22204c897cc0d91 P 599bfef32939446d93be83406a84d8bf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:36:09.297755 32362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:09.297500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:43889 (local address 127.16.157.66:41899)
0928 06:36:09.297682 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:09.297716 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:43889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":127,"threads_started":1}
I20260928 06:36:09.427820 32363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:09.427578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:43889 (local address 127.16.157.67:47241)
0928 06:36:09.427720 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:09.427760 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:43889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":76,"threads_started":1}
I20260928 06:36:10.249053 32364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:10.248824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:43889 (local address 127.16.157.65:41259)
0928 06:36:10.248957 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:10.248997 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:43889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":77,"threads_started":1}
I20260928 06:36:10.298416 32365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:10.298189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:43889 (local address 127.16.157.66:53739)
0928 06:36:10.298327 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:10.298371 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:43889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":65,"threads_started":1}
I20260928 06:36:10.428509 32366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:10.428265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:43889 (local address 127.16.157.67:36599)
0928 06:36:10.428397 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:10.428453 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:43889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":60,"threads_started":1}
I20260928 06:36:11.249878 32367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:11.249640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:43889 (local address 127.16.157.65:39175)
0928 06:36:11.249794 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:11.249831 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:43889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":80,"threads_started":1}
I20260928 06:36:11.299134 32368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:11.298849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:43889 (local address 127.16.157.66:56905)
0928 06:36:11.299028 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:11.299070 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:43889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":73,"threads_started":1}
I20260928 06:36:11.429155 32369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:11.428939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:43889 (local address 127.16.157.67:47401)
0928 06:36:11.429078 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:11.429112 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:43889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":81,"threads_started":1}
I20260928 06:36:12.250797 32370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:12.250541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:43889 (local address 127.16.157.65:39813)
0928 06:36:12.250705 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:12.250749 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:43889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":83,"threads_started":1}
I20260928 06:36:12.299906 32371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:12.299639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:43889 (local address 127.16.157.66:35831)
0928 06:36:12.299799 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:12.299854 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:43889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":59,"threads_started":1}
I20260928 06:36:12.430061 32372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:12.429840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:43889 (local address 127.16.157.67:55683)
0928 06:36:12.429971 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:12.430012 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:43889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":64,"threads_started":1}
I20260928 06:36:13.251559 32373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:13.251347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:43889 (local address 127.16.157.65:52261)
0928 06:36:13.251477 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:13.251511 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:43889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":74,"threads_started":1}
I20260928 06:36:13.300607 32374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:13.300362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:43889 (local address 127.16.157.66:36445)
0928 06:36:13.300516 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:13.300558 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:43889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":84,"threads_started":1}
I20260928 06:36:13.430666 32375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:13.430444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:43889 (local address 127.16.157.67:56803)
0928 06:36:13.430580 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:13.430625 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:43889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":56,"threads_started":1}
I20260928 06:36:14.252228 32376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:14.251988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:43889 (local address 127.16.157.65:57707)
0928 06:36:14.252131 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:14.252182 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:43889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":78,"threads_started":1}
I20260928 06:36:14.301229 32377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:14.301016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:43889 (local address 127.16.157.66:46307)
0928 06:36:14.301149 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:14.301185 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:43889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":56,"threads_started":1}
I20260928 06:36:14.431299 32378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:14.431090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:43889 (local address 127.16.157.67:57647)
0928 06:36:14.431223 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:14.431258 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:43889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":75,"threads_started":1}
I20260928 06:36:15.252907 32379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:15.252695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:43889 (local address 127.16.157.65:38739)
0928 06:36:15.252828 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:15.252859 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:43889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":77,"threads_started":1}
I20260928 06:36:15.301898 32380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:15.301693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:43889 (local address 127.16.157.66:37577)
0928 06:36:15.301828 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:15.301860 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:43889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":75,"threads_started":1}
I20260928 06:36:15.427059 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 31901
I20260928 06:36:15.432001 32381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:15.431783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:43889 (local address 127.16.157.67:56081)
0928 06:36:15.431914 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:15.431945 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:43889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":76,"threads_started":1}
I20260928 06:36:15.432441 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 32064
I20260928 06:36:15.437482 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 32197
2026-09-28T06:36:15Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18283 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:36:15 dist-test-slave-rrj5 krb5kdc[32384](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:36:15 dist-test-slave-rrj5 krb5kdc[32384](info): set up 2 sockets
Sep 28 06:36:15 dist-test-slave-rrj5 krb5kdc[32384](info): commencing operation
krb5kdc: starting...
W20260928 06:36:16.482481 17013 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Sep 28 06:36:16 dist-test-slave-rrj5 krb5kdc[32384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:36:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:36:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.157.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab.
Entry for principal kudu/127.16.157.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab.
I20260928 06:36:16.596589 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:33113
--webserver_interface=127.16.157.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=kudu/127.16.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:43161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.157.126:33113
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:36:16.675523 32397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:36:16.675715 32397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:36:16.675737 32397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:36:16.676874 32397 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 06:36:16.676914 32397 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 06:36:16.676927 32397 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 06:36:16.676939 32397 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:36:16.676980 32397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:36:16.677008 32397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:36:16.677026 32397 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:36:16.677052 32397 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:36:16.678483 32397 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:43161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.16.157.126:33113
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=kudu/127.16.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:33113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:36:16.678705 32397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:36:16.678930 32397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:36:16.684226 32405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:16.684226 32403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:16.684281 32397 server_base.cc:1034] running on GCE node
W20260928 06:36:16.684245 32402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:16.684700 32397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:36:16.684994 32397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:36:16.686158 32397 hybrid_clock.cc:648] HybridClock initialized: now 1790577376686151 us; error 36 us; skew 500 ppm
Sep 28 06:36:16 dist-test-slave-rrj5 krb5kdc[32384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577376, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:36:16.688625 32397 init.cc:377] Logged in from keytab as kudu/127.16.157.126@KRBTEST.COM (short username kudu)
I20260928 06:36:16.689044 32397 webserver.cc:460] Webserver started at http://127.16.157.126:40755/ using document root <none> and password file <none>
I20260928 06:36:16.689271 32397 fs_manager.cc:359] Metadata directory not provided
I20260928 06:36:16.689337 32397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:36:16.689491 32397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:36:16.690563 32397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/master-0/data/instance:
uuid: "aae1e59bc3dd4b2c9599a522cbf5b565"
format_stamp: "Formatted at 2026-09-28 06:36:16 on dist-test-slave-rrj5"
server_key: "135668f5271aa01d9c65290e12cab450"
server_key_iv: "e688892c5d0ed37454c2914e627fcf4c"
server_key_version: "encryptionkey@0"
I20260928 06:36:16.690924 32397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/master-0/wal/instance:
uuid: "aae1e59bc3dd4b2c9599a522cbf5b565"
format_stamp: "Formatted at 2026-09-28 06:36:16 on dist-test-slave-rrj5"
server_key: "135668f5271aa01d9c65290e12cab450"
server_key_iv: "e688892c5d0ed37454c2914e627fcf4c"
server_key_version: "encryptionkey@0"
I20260928 06:36:16.692257 32397 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:36:16.692975 32412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:16.693141 32397 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:36:16.693241 32397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/master-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/master-0/wal
uuid: "aae1e59bc3dd4b2c9599a522cbf5b565"
format_stamp: "Formatted at 2026-09-28 06:36:16 on dist-test-slave-rrj5"
server_key: "135668f5271aa01d9c65290e12cab450"
server_key_iv: "e688892c5d0ed37454c2914e627fcf4c"
server_key_version: "encryptionkey@0"
I20260928 06:36:16.693322 32397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:36:16.709844 32397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:36:16.711972 32397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:36:16.712144 32397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:36:16.716310 32397 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.126:33113
I20260928 06:36:16.716343 32464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.126:33113 every 8 connection(s)
I20260928 06:36:16.716679 32397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
I20260928 06:36:16.717526 32465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:36:16.719787 32465 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aae1e59bc3dd4b2c9599a522cbf5b565: Bootstrap starting.
I20260928 06:36:16.720357 32465 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aae1e59bc3dd4b2c9599a522cbf5b565: Neither blocks nor log segments found. Creating new log.
I20260928 06:36:16.720666 32465 log.cc:824] T 00000000000000000000000000000000 P aae1e59bc3dd4b2c9599a522cbf5b565: Log is configured to *not* fsync() on all Append() calls
I20260928 06:36:16.720921 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 32397
I20260928 06:36:16.721045 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/master-0/wal/instance
I20260928 06:36:16.721254 17013 external_mini_cluster.cc:1428] Setting key 397c42df0d308a37b64f032438e09e7a
I20260928 06:36:16.721441 32465 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aae1e59bc3dd4b2c9599a522cbf5b565: No bootstrap required, opened a new log
I20260928 06:36:16.723269 32465 raft_consensus.cc:348] T 00000000000000000000000000000000 P aae1e59bc3dd4b2c9599a522cbf5b565 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aae1e59bc3dd4b2c9599a522cbf5b565" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 33113 } }
I20260928 06:36:16.723428 32465 raft_consensus.cc:374] T 00000000000000000000000000000000 P aae1e59bc3dd4b2c9599a522cbf5b565 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:36:16.723464 32465 raft_consensus.cc:729] T 00000000000000000000000000000000 P aae1e59bc3dd4b2c9599a522cbf5b565 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae1e59bc3dd4b2c9599a522cbf5b565, State: Initialized, Role: FOLLOWER
I20260928 06:36:16.723625 32465 consensus_queue.cc:260] T 00000000000000000000000000000000 P aae1e59bc3dd4b2c9599a522cbf5b565 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aae1e59bc3dd4b2c9599a522cbf5b565" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 33113 } }
I20260928 06:36:16.723716 32465 raft_consensus.cc:388] T 00000000000000000000000000000000 P aae1e59bc3dd4b2c9599a522cbf5b565 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:36:16.723753 32465 raft_consensus.cc:482] T 00000000000000000000000000000000 P aae1e59bc3dd4b2c9599a522cbf5b565 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:36:16.723803 32465 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aae1e59bc3dd4b2c9599a522cbf5b565 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:36:16.724756 32465 raft_consensus.cc:504] T 00000000000000000000000000000000 P aae1e59bc3dd4b2c9599a522cbf5b565 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aae1e59bc3dd4b2c9599a522cbf5b565" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 33113 } }
I20260928 06:36:16.724839 32465 leader_election.cc:302] T 00000000000000000000000000000000 P aae1e59bc3dd4b2c9599a522cbf5b565 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aae1e59bc3dd4b2c9599a522cbf5b565; no voters:
I20260928 06:36:16.725005 32465 leader_election.cc:288] T 00000000000000000000000000000000 P aae1e59bc3dd4b2c9599a522cbf5b565 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:36:16.725041 32470 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aae1e59bc3dd4b2c9599a522cbf5b565 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:36:16.725155 32470 raft_consensus.cc:686] T 00000000000000000000000000000000 P aae1e59bc3dd4b2c9599a522cbf5b565 [term 1 LEADER]: Becoming Leader. State: Replica: aae1e59bc3dd4b2c9599a522cbf5b565, State: Running, Role: LEADER
I20260928 06:36:16.725276 32465 sys_catalog.cc:564] T 00000000000000000000000000000000 P aae1e59bc3dd4b2c9599a522cbf5b565 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:36:16.725294 32470 consensus_queue.cc:237] T 00000000000000000000000000000000 P aae1e59bc3dd4b2c9599a522cbf5b565 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aae1e59bc3dd4b2c9599a522cbf5b565" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 33113 } }
I20260928 06:36:16.725740 32471 sys_catalog.cc:455] T 00000000000000000000000000000000 P aae1e59bc3dd4b2c9599a522cbf5b565 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aae1e59bc3dd4b2c9599a522cbf5b565" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aae1e59bc3dd4b2c9599a522cbf5b565" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 33113 } } }
I20260928 06:36:16.725749 32472 sys_catalog.cc:455] T 00000000000000000000000000000000 P aae1e59bc3dd4b2c9599a522cbf5b565 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aae1e59bc3dd4b2c9599a522cbf5b565. Latest consensus state: current_term: 1 leader_uuid: "aae1e59bc3dd4b2c9599a522cbf5b565" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aae1e59bc3dd4b2c9599a522cbf5b565" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 33113 } } }
I20260928 06:36:16.725935 32471 sys_catalog.cc:458] T 00000000000000000000000000000000 P aae1e59bc3dd4b2c9599a522cbf5b565 [sys.catalog]: This master's current role is: LEADER
I20260928 06:36:16.725946 32472 sys_catalog.cc:458] T 00000000000000000000000000000000 P aae1e59bc3dd4b2c9599a522cbf5b565 [sys.catalog]: This master's current role is: LEADER
I20260928 06:36:16.726397 32479 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:36:16.727093 32479 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:36:16.728602 32479 catalog_manager.cc:1347] Generated new cluster ID: 40c2479ccc80479fa303215377f5c4eb
I20260928 06:36:16.728670 32479 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:36:16 dist-test-slave-rrj5 krb5kdc[32384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:36:16.734246 32469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:16.721799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46430 (local address 127.16.157.126:33113)
0928 06:36:16.721967 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:16.721975 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:16.722010 (+ 35us) server_negotiation.cc:407] Connection header received
0928 06:36:16.723247 (+ 1237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:16.723253 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:16.723305 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:16.723413 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:16.724240 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:16.725391 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:16.726543 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:16.727158 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:16.727680 (+ 522us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:16.730109 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:16.730113 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:16.730114 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:16.730124 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:16.732000 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:16.732411 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:16.732414 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:16.732414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:16.732855 (+ 441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:16.733092 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:16.733093 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:16.733093 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:16.733363 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:16.733481 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:16.733889 (+ 408us) server_negotiation.cc:299] Negotiation successful
0928 06:36:16.734048 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":95,"threads_started":1}
I20260928 06:36:16.735913 32479 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:36:16.736393 32479 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:36:16.739550 32479 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aae1e59bc3dd4b2c9599a522cbf5b565: Generated new TSK 0
I20260928 06:36:16.739722 32479 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.157.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab.
Entry for principal kudu/127.16.157.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab.
I20260928 06:36:16.764952 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.65:0
--local_ip_for_outbound_sockets=127.16.157.65
--webserver_interface=127.16.157.65
--webserver_port=0
--tserver_master_addrs=127.16.157.126:33113
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab
--principal=kudu/127.16.157.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:43161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:36:16.846011 32491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:36:16.846237 32491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:36:16.846261 32491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:36:16.847412 32491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:36:16.847455 32491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:36:16.847481 32491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.65
I20260928 06:36:16.848994 32491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:43161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab
--principal=kudu/127.16.157.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.157.65
--webserver_port=0
--tserver_master_addrs=127.16.157.126:33113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32491
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.65
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:36:16.849216 32491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:36:16.849467 32491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:36:16.854485 32499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:16.854454 32496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:16.854463 32497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:16.854725 32491 server_base.cc:1034] running on GCE node
I20260928 06:36:16.854913 32491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:36:16.855110 32491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:36:16.856273 32491 hybrid_clock.cc:648] HybridClock initialized: now 1790577376856257 us; error 27 us; skew 500 ppm
Sep 28 06:36:16 dist-test-slave-rrj5 krb5kdc[32384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577376, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:36:16.858922 32491 init.cc:377] Logged in from keytab as kudu/127.16.157.65@KRBTEST.COM (short username kudu)
I20260928 06:36:16.859345 32491 webserver.cc:460] Webserver started at http://127.16.157.65:35837/ using document root <none> and password file <none>
I20260928 06:36:16.859589 32491 fs_manager.cc:359] Metadata directory not provided
I20260928 06:36:16.859660 32491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:36:16.859802 32491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:36:16.860810 32491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-0/data/instance:
uuid: "e0f3d400343544cc93a22d3c756e7838"
format_stamp: "Formatted at 2026-09-28 06:36:16 on dist-test-slave-rrj5"
server_key: "efcde22511bade5ec702202e732afb3e"
server_key_iv: "fbe53357b50bc140b9b6685798859930"
server_key_version: "encryptionkey@0"
I20260928 06:36:16.861167 32491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-0/wal/instance:
uuid: "e0f3d400343544cc93a22d3c756e7838"
format_stamp: "Formatted at 2026-09-28 06:36:16 on dist-test-slave-rrj5"
server_key: "efcde22511bade5ec702202e732afb3e"
server_key_iv: "fbe53357b50bc140b9b6685798859930"
server_key_version: "encryptionkey@0"
I20260928 06:36:16.862565 32491 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:36:16.863267 32506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:16.863438 32491 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:36:16.863548 32491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-0/wal
uuid: "e0f3d400343544cc93a22d3c756e7838"
format_stamp: "Formatted at 2026-09-28 06:36:16 on dist-test-slave-rrj5"
server_key: "efcde22511bade5ec702202e732afb3e"
server_key_iv: "fbe53357b50bc140b9b6685798859930"
server_key_version: "encryptionkey@0"
I20260928 06:36:16.863622 32491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:36:16.892468 32491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:36:16.894380 32491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:36:16.894548 32491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:36:16.894790 32491 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:36:16.895092 32491 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:36:16.895146 32491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:16.895190 32491 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:36:16.895215 32491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:16.901680 32491 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.65:45055
I20260928 06:36:16.901800 32619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.65:45055 every 8 connection(s)
I20260928 06:36:16.902096 32491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
Sep 28 06:36:16 dist-test-slave-rrj5 krb5kdc[32384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577376, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.65@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:36:16.909922 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 32491
I20260928 06:36:16.910022 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-0/wal/instance
I20260928 06:36:16.910255 17013 external_mini_cluster.cc:1428] Setting key c5e7c80f3b90f474ed280a045900d114
I20260928 06:36:16.913414 32622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:16.902846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:33113 (local address 127.16.157.65:38813)
0928 06:36:16.903031 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:16.903090 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:16.903851 (+ 761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:16.904197 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:16.904201 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:16.905019 (+ 818us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:16.905344 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:16.905350 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:16.906274 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:16.906275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:16.907224 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:16.907227 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:16.907646 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:16.907647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:16.907786 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:16.908322 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:16.908331 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:16.910309 (+ 1978us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:36:16.912010 (+ 1701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:16.912017 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:16.912020 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:16.912350 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:16.912594 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:16.912594 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:16.912595 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:16.912694 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:16.912964 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:36:16.912966 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:36:16.913079 (+ 113us) client_negotiation.cc:769] Sending connection context
0928 06:36:16.913154 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 06:36:16.913265 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":89,"threads_started":1}
I20260928 06:36:16.913414 32469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:16.902867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:38813 (local address 127.16.157.126:33113)
0928 06:36:16.902936 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:16.902938 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:16.903206 (+ 268us) server_negotiation.cc:407] Connection header received
0928 06:36:16.904007 (+ 801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:16.904008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:16.904023 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:16.904107 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:16.905461 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:16.906177 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:16.907327 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:16.907561 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:16.907636 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:16.910413 (+ 2777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:16.910415 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:16.910415 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:16.910427 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:16.911914 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:16.912436 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:16.912438 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:16.912438 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:16.912509 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:16.912758 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:16.912759 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:16.912759 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:16.912903 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:16.912979 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:16.913201 (+ 222us) server_negotiation.cc:299] Negotiation successful
0928 06:36:16.913303 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 06:36:16.913980 32620 heartbeater.cc:347] Connected to a master server at 127.16.157.126:33113
I20260928 06:36:16.914083 32620 heartbeater.cc:464] Registering TS with master...
I20260928 06:36:16.914271 32620 heartbeater.cc:511] Master 127.16.157.126:33113 requested a full tablet report, sending...
I20260928 06:36:16.914821 32429 ts_manager.cc:194] Registered new tserver with Master: e0f3d400343544cc93a22d3c756e7838 (127.16.157.65:45055)
I20260928 06:36:16.915445 32429 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.157.65@KRBTEST.COM'} at 127.16.157.65:38813
WARNING: no policy specified for kudu/127.16.157.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab.
Entry for principal kudu/127.16.157.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab.
I20260928 06:36:16.935771 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.66:0
--local_ip_for_outbound_sockets=127.16.157.66
--webserver_interface=127.16.157.66
--webserver_port=0
--tserver_master_addrs=127.16.157.126:33113
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab
--principal=kudu/127.16.157.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:43161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:36:17.016569 32625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:36:17.016788 32625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:36:17.016822 32625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:36:17.017941 32625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:36:17.018008 32625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:36:17.018059 32625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.66
I20260928 06:36:17.019470 32625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:43161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab
--principal=kudu/127.16.157.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.157.66
--webserver_port=0
--tserver_master_addrs=127.16.157.126:33113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.66
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:36:17.019726 32625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:36:17.019970 32625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:36:17.025172 32631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:17.025220 32625 server_base.cc:1034] running on GCE node
W20260928 06:36:17.025229 32633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:17.025192 32630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:17.025624 32625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:36:17.025895 32625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:36:17.027058 32625 hybrid_clock.cc:648] HybridClock initialized: now 1790577377027037 us; error 34 us; skew 500 ppm
Sep 28 06:36:17 dist-test-slave-rrj5 krb5kdc[32384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577377, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:36:17.029495 32625 init.cc:377] Logged in from keytab as kudu/127.16.157.66@KRBTEST.COM (short username kudu)
I20260928 06:36:17.029994 32625 webserver.cc:460] Webserver started at http://127.16.157.66:34285/ using document root <none> and password file <none>
I20260928 06:36:17.030227 32625 fs_manager.cc:359] Metadata directory not provided
I20260928 06:36:17.030293 32625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:36:17.030442 32625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:36:17.031446 32625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-1/data/instance:
uuid: "a7d3e2f7b2d8486fbd3c67a10f6f9447"
format_stamp: "Formatted at 2026-09-28 06:36:17 on dist-test-slave-rrj5"
server_key: "b32ea6555f8b48a6cdc0d85f0038d664"
server_key_iv: "e4234dfc0b751e95b47b1c06c45088e9"
server_key_version: "encryptionkey@0"
I20260928 06:36:17.031800 32625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-1/wal/instance:
uuid: "a7d3e2f7b2d8486fbd3c67a10f6f9447"
format_stamp: "Formatted at 2026-09-28 06:36:17 on dist-test-slave-rrj5"
server_key: "b32ea6555f8b48a6cdc0d85f0038d664"
server_key_iv: "e4234dfc0b751e95b47b1c06c45088e9"
server_key_version: "encryptionkey@0"
I20260928 06:36:17.033080 32625 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 06:36:17.033792 32640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:17.033949 32625 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:36:17.034056 32625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-1/wal
uuid: "a7d3e2f7b2d8486fbd3c67a10f6f9447"
format_stamp: "Formatted at 2026-09-28 06:36:17 on dist-test-slave-rrj5"
server_key: "b32ea6555f8b48a6cdc0d85f0038d664"
server_key_iv: "e4234dfc0b751e95b47b1c06c45088e9"
server_key_version: "encryptionkey@0"
I20260928 06:36:17.034128 32625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:36:17.050376 32625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:36:17.052415 32625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:36:17.052582 32625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:36:17.052802 32625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:36:17.053109 32625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:36:17.053143 32625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:17.053179 32625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:36:17.053223 32625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:17.059141 32625 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.66:34633
I20260928 06:36:17.059218 32753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.66:34633 every 8 connection(s)
I20260928 06:36:17.059540 32625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
I20260928 06:36:17.060189 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 32625
I20260928 06:36:17.060297 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-1/wal/instance
I20260928 06:36:17.060511 17013 external_mini_cluster.cc:1428] Setting key 99048c7f75a1628ce7eaf2752a12fc4e
Sep 28 06:36:17 dist-test-slave-rrj5 krb5kdc[32384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577377, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.66@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.157.67@KRBTEST.COM; defaulting to no policy
I20260928 06:36:17.069495 32469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:17.060301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.66:55037 (local address 127.16.157.126:33113)
0928 06:36:17.060359 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:17.060361 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:17.060565 (+ 204us) server_negotiation.cc:407] Connection header received
0928 06:36:17.061291 (+ 726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:17.061292 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:17.061314 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:17.061349 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:17.062345 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:17.062864 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:17.063845 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:17.064071 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:17.064159 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:17.066537 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:17.066539 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:17.066539 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:17.066547 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:17.068042 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:17.068470 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:17.068471 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:17.068471 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:17.068537 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:17.068799 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:17.068801 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:17.068801 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:17.068957 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:17.069038 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:17.069314 (+ 276us) server_negotiation.cc:299] Negotiation successful
0928 06:36:17.069405 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 06:36:17.069716 32756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:17.060282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:33113 (local address 127.16.157.66:55037)
0928 06:36:17.060441 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:17.060479 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:17.061189 (+ 710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:17.061410 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:17.061413 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:17.062059 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:17.062246 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:17.062250 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:17.062959 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:17.062968 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:17.063754 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:17.063756 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:17.064173 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:17.064174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:17.064303 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:17.064927 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:17.064943 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:17.066426 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:36:17.068152 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:17.068161 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:17.068163 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:17.068391 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:17.068607 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:17.068608 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:17.068609 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:17.068719 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:17.069029 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:36:17.069036 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:36:17.069178 (+ 142us) client_negotiation.cc:769] Sending connection context
0928 06:36:17.069281 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 06:36:17.069440 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":93,"threads_started":1}
I20260928 06:36:17.070574 32754 heartbeater.cc:347] Connected to a master server at 127.16.157.126:33113
I20260928 06:36:17.070686 32754 heartbeater.cc:464] Registering TS with master...
I20260928 06:36:17.070928 32754 heartbeater.cc:511] Master 127.16.157.126:33113 requested a full tablet report, sending...
I20260928 06:36:17.071336 32429 ts_manager.cc:194] Registered new tserver with Master: a7d3e2f7b2d8486fbd3c67a10f6f9447 (127.16.157.66:34633)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.67@KRBTEST.COM" created.
I20260928 06:36:17.071759 32429 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.157.66@KRBTEST.COM'} at 127.16.157.66:55037
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab.
Entry for principal kudu/127.16.157.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab.
I20260928 06:36:17.084455 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.67:0
--local_ip_for_outbound_sockets=127.16.157.67
--webserver_interface=127.16.157.67
--webserver_port=0
--tserver_master_addrs=127.16.157.126:33113
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab
--principal=kudu/127.16.157.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:43161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:36:17.164707 32759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:36:17.164896 32759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:36:17.164918 32759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:36:17.166256 32759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:36:17.166299 32759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:36:17.166348 32759 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.67
I20260928 06:36:17.167866 32759 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:43161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab
--principal=kudu/127.16.157.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.157.67
--webserver_port=0
--tserver_master_addrs=127.16.157.126:33113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.67
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:36:17.168102 32759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:36:17.168330 32759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:36:17.173135 32767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:17.173336 32759 server_base.cc:1034] running on GCE node
W20260928 06:36:17.173132 32765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:17.173247 32764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:17.173647 32759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:36:17.173852 32759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:36:17.175017 32759 hybrid_clock.cc:648] HybridClock initialized: now 1790577377174986 us; error 47 us; skew 500 ppm
Sep 28 06:36:17 dist-test-slave-rrj5 krb5kdc[32384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577377, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:36:17.177709 32759 init.cc:377] Logged in from keytab as kudu/127.16.157.67@KRBTEST.COM (short username kudu)
I20260928 06:36:17.178133 32759 webserver.cc:460] Webserver started at http://127.16.157.67:36887/ using document root <none> and password file <none>
I20260928 06:36:17.178375 32759 fs_manager.cc:359] Metadata directory not provided
I20260928 06:36:17.178445 32759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:36:17.178623 32759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:36:17.179653 32759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-2/data/instance:
uuid: "17f0c1fd75e8472680dc5e3148a9c42b"
format_stamp: "Formatted at 2026-09-28 06:36:17 on dist-test-slave-rrj5"
server_key: "8a201b81bfc149b840eba2129582b3e7"
server_key_iv: "42b053b2b13010f5bfed8243f82a9c2b"
server_key_version: "encryptionkey@0"
I20260928 06:36:17.180020 32759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-2/wal/instance:
uuid: "17f0c1fd75e8472680dc5e3148a9c42b"
format_stamp: "Formatted at 2026-09-28 06:36:17 on dist-test-slave-rrj5"
server_key: "8a201b81bfc149b840eba2129582b3e7"
server_key_iv: "42b053b2b13010f5bfed8243f82a9c2b"
server_key_version: "encryptionkey@0"
I20260928 06:36:17.181317 32759 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:36:17.182111 306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:17.182281 32759 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:17.182381 32759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-2/wal
uuid: "17f0c1fd75e8472680dc5e3148a9c42b"
format_stamp: "Formatted at 2026-09-28 06:36:17 on dist-test-slave-rrj5"
server_key: "8a201b81bfc149b840eba2129582b3e7"
server_key_iv: "42b053b2b13010f5bfed8243f82a9c2b"
server_key_version: "encryptionkey@0"
I20260928 06:36:17.182463 32759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:36:17.206363 32759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:36:17.208164 32759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:36:17.208321 32759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:36:17.208551 32759 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:36:17.208843 32759 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:36:17.208896 32759 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:17.208937 32759 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:36:17.208967 32759 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:17.215495 32759 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.67:33161
I20260928 06:36:17.215545 419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.67:33161 every 8 connection(s)
I20260928 06:36:17.215860 32759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
I20260928 06:36:17.219096 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 32759
I20260928 06:36:17.219182 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-2/wal/instance
I20260928 06:36:17.219372 17013 external_mini_cluster.cc:1428] Setting key a00a31ab95eb63926ac18838bfa899cd
Sep 28 06:36:17 dist-test-slave-rrj5 krb5kdc[32384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577377, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.67@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:36:17.225153 32469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:17.216500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:38535 (local address 127.16.157.126:33113)
0928 06:36:17.216570 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:17.216577 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:17.216788 (+ 211us) server_negotiation.cc:407] Connection header received
0928 06:36:17.217294 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:17.217295 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:17.217304 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:17.217354 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:17.218301 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:17.218774 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:17.220077 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:17.220353 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:17.220424 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:17.222531 (+ 2107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:17.222532 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:17.222532 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:17.222539 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:17.224028 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:17.224401 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:17.224402 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:17.224403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:17.224454 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:17.224671 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:17.224671 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:17.224672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:17.224771 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:17.224822 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:17.225028 (+ 206us) server_negotiation.cc:299] Negotiation successful
0928 06:36:17.225091 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 06:36:17.225358 422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:17.216482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:33113 (local address 127.16.157.67:38535)
0928 06:36:17.216655 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:17.216695 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:17.217209 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:17.217413 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:17.217416 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:17.218033 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:17.218222 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:17.218228 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:17.218860 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:17.218860 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:17.219995 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:17.219998 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:17.220446 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:17.220447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:17.220560 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:17.220996 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:17.221003 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:17.222445 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:36:17.224120 (+ 1675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:17.224126 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:17.224128 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:17.224337 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:17.224511 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:17.224512 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:17.224512 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:17.224617 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:17.224811 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:36:17.224813 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:36:17.224925 (+ 112us) client_negotiation.cc:769] Sending connection context
0928 06:36:17.225001 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 06:36:17.225121 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":97,"threads_started":1}
I20260928 06:36:17.225952 420 heartbeater.cc:347] Connected to a master server at 127.16.157.126:33113
I20260928 06:36:17.226071 420 heartbeater.cc:464] Registering TS with master...
I20260928 06:36:17.226269 420 heartbeater.cc:511] Master 127.16.157.126:33113 requested a full tablet report, sending...
I20260928 06:36:17.226665 32429 ts_manager.cc:194] Registered new tserver with Master: 17f0c1fd75e8472680dc5e3148a9c42b (127.16.157.67:33161)
I20260928 06:36:17.227082 32429 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.157.67@KRBTEST.COM'} at 127.16.157.67:38535
I20260928 06:36:17.231073 17013 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:36:17.238164 32469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:17.232486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46440 (local address 127.16.157.126:33113)
0928 06:36:17.232537 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:17.232539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:17.232631 (+ 92us) server_negotiation.cc:407] Connection header received
0928 06:36:17.232668 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:17.232669 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:17.232677 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:17.232721 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:17.233250 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:17.233741 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:17.234507 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:17.234682 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:17.234727 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:17.235757 (+ 1030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:17.235758 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:17.235758 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:17.235764 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:17.237082 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:17.237410 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:17.237411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:17.237411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:17.237460 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:17.237697 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:17.237697 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:17.237698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:17.237872 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:17.237949 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:17.238051 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 06:36:17.238107 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 06:36:17.239605 32429 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46440:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:36:17.241174 32429 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:36:17.247974 432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:17.243679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.67:33161 (local address 127.0.0.1:59434)
0928 06:36:17.243836 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:17.243885 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:17.243957 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:17.244482 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:17.244485 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:17.244488 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:36:17.244576 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:17.244579 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:17.245473 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:17.245474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:17.246641 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:17.246644 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:17.247664 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:17.247665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:17.247753 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:17.247766 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 06:36:17.247840 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 06:36:17.247901 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":2,"thread_start_us":66,"threads_started":1}
I20260928 06:36:17.248021 434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:17.243856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59434 (local address 127.16.157.67:33161)
0928 06:36:17.244283 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:17.244286 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:17.244294 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:36:17.244326 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:17.244328 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:17.244359 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:17.244664 (+ 305us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:36:17.244723 (+ 59us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:17.245373 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:17.246732 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:17.247572 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:17.247653 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:17.247706 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:17.247883 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 06:36:17.247926 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":104,"threads_started":1}
I20260928 06:36:17.248219 431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:17.243679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.66:34633 (local address 127.0.0.1:42488)
0928 06:36:17.244000 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:17.244010 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:17.244063 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:17.244488 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:17.244488 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:17.244490 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:36:17.244576 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:17.244578 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:17.246067 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:17.246068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:17.246978 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:17.246980 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:17.248000 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:17.248001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:17.248080 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:17.248086 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 06:36:17.248119 (+ 33us) client_negotiation.cc:240] Negotiation successful
0928 06:36:17.248158 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":20,"thread_start_us":80,"threads_started":1}
I20260928 06:36:17.248504 436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:17.244068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42488 (local address 127.16.157.66:34633)
0928 06:36:17.244263 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:17.244268 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:17.244279 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:36:17.244324 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:17.244326 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:17.244343 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:17.244427 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:36:17.244745 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:17.245937 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:17.247073 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:17.247903 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:17.248271 (+ 368us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:17.248316 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:17.248344 (+ 28us) server_negotiation.cc:299] Negotiation successful
0928 06:36:17.248398 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":117,"threads_started":1}
I20260928 06:36:17.249078 433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:17.243882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.65:45055 (local address 127.0.0.1:45518)
0928 06:36:17.244287 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:17.244297 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:17.244419 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:17.244683 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:17.244684 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:17.244686 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:36:17.244737 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:17.244739 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:17.247066 (+ 2327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:17.247066 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:17.247826 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:17.247828 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:17.248829 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:17.248830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:17.248887 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:17.248891 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:36:17.248946 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 06:36:17.248983 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":68,"threads_started":1}
I20260928 06:36:17.249060 354 tablet_service.cc:1467] Processing CreateTablet for tablet 1c2c7375029a4839bad10b59008830f9 (DEFAULT_TABLE table=test-table [id=be71b3c386074100910cbb3fb4f024bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:36:17.249119 435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:17.244037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45518 (local address 127.16.157.65:45055)
0928 06:36:17.244222 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:17.244227 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:17.244365 (+ 138us) server_negotiation.cc:407] Connection header received
0928 06:36:17.244529 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:17.244532 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:17.244548 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:17.244618 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:36:17.244863 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:17.245947 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:17.247913 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:17.248683 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:17.248747 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:17.248795 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:17.248977 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 06:36:17.249025 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":100,"threads_started":1}
I20260928 06:36:17.249167 32688 tablet_service.cc:1467] Processing CreateTablet for tablet 1c2c7375029a4839bad10b59008830f9 (DEFAULT_TABLE table=test-table [id=be71b3c386074100910cbb3fb4f024bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:36:17.249436 354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c2c7375029a4839bad10b59008830f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:36:17.249459 32688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c2c7375029a4839bad10b59008830f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:36:17.249899 32554 tablet_service.cc:1467] Processing CreateTablet for tablet 1c2c7375029a4839bad10b59008830f9 (DEFAULT_TABLE table=test-table [id=be71b3c386074100910cbb3fb4f024bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:36:17.250218 32554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c2c7375029a4839bad10b59008830f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:36:17.252460 437 tablet_bootstrap.cc:492] T 1c2c7375029a4839bad10b59008830f9 P 17f0c1fd75e8472680dc5e3148a9c42b: Bootstrap starting.
I20260928 06:36:17.253099 437 tablet_bootstrap.cc:654] T 1c2c7375029a4839bad10b59008830f9 P 17f0c1fd75e8472680dc5e3148a9c42b: Neither blocks nor log segments found. Creating new log.
I20260928 06:36:17.253376 438 tablet_bootstrap.cc:492] T 1c2c7375029a4839bad10b59008830f9 P a7d3e2f7b2d8486fbd3c67a10f6f9447: Bootstrap starting.
I20260928 06:36:17.253762 437 log.cc:824] T 1c2c7375029a4839bad10b59008830f9 P 17f0c1fd75e8472680dc5e3148a9c42b: Log is configured to *not* fsync() on all Append() calls
I20260928 06:36:17.253947 439 tablet_bootstrap.cc:492] T 1c2c7375029a4839bad10b59008830f9 P e0f3d400343544cc93a22d3c756e7838: Bootstrap starting.
I20260928 06:36:17.254122 438 tablet_bootstrap.cc:654] T 1c2c7375029a4839bad10b59008830f9 P a7d3e2f7b2d8486fbd3c67a10f6f9447: Neither blocks nor log segments found. Creating new log.
I20260928 06:36:17.254544 437 tablet_bootstrap.cc:492] T 1c2c7375029a4839bad10b59008830f9 P 17f0c1fd75e8472680dc5e3148a9c42b: No bootstrap required, opened a new log
I20260928 06:36:17.254565 439 tablet_bootstrap.cc:654] T 1c2c7375029a4839bad10b59008830f9 P e0f3d400343544cc93a22d3c756e7838: Neither blocks nor log segments found. Creating new log.
I20260928 06:36:17.254619 437 ts_tablet_manager.cc:1397] T 1c2c7375029a4839bad10b59008830f9 P 17f0c1fd75e8472680dc5e3148a9c42b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 06:36:17.254719 438 log.cc:824] T 1c2c7375029a4839bad10b59008830f9 P a7d3e2f7b2d8486fbd3c67a10f6f9447: Log is configured to *not* fsync() on all Append() calls
I20260928 06:36:17.255187 439 log.cc:824] T 1c2c7375029a4839bad10b59008830f9 P e0f3d400343544cc93a22d3c756e7838: Log is configured to *not* fsync() on all Append() calls
I20260928 06:36:17.255505 438 tablet_bootstrap.cc:492] T 1c2c7375029a4839bad10b59008830f9 P a7d3e2f7b2d8486fbd3c67a10f6f9447: No bootstrap required, opened a new log
I20260928 06:36:17.255590 438 ts_tablet_manager.cc:1397] T 1c2c7375029a4839bad10b59008830f9 P a7d3e2f7b2d8486fbd3c67a10f6f9447: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:36:17.256326 439 tablet_bootstrap.cc:492] T 1c2c7375029a4839bad10b59008830f9 P e0f3d400343544cc93a22d3c756e7838: No bootstrap required, opened a new log
I20260928 06:36:17.256440 439 ts_tablet_manager.cc:1397] T 1c2c7375029a4839bad10b59008830f9 P e0f3d400343544cc93a22d3c756e7838: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 06:36:17.259153 438 raft_consensus.cc:348] T 1c2c7375029a4839bad10b59008830f9 P a7d3e2f7b2d8486fbd3c67a10f6f9447 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f0c1fd75e8472680dc5e3148a9c42b" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 33161 } } peers { permanent_uuid: "a7d3e2f7b2d8486fbd3c67a10f6f9447" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 34633 } } peers { permanent_uuid: "e0f3d400343544cc93a22d3c756e7838" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 45055 } }
I20260928 06:36:17.259196 439 raft_consensus.cc:348] T 1c2c7375029a4839bad10b59008830f9 P e0f3d400343544cc93a22d3c756e7838 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f0c1fd75e8472680dc5e3148a9c42b" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 33161 } } peers { permanent_uuid: "a7d3e2f7b2d8486fbd3c67a10f6f9447" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 34633 } } peers { permanent_uuid: "e0f3d400343544cc93a22d3c756e7838" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 45055 } }
I20260928 06:36:17.259352 438 raft_consensus.cc:374] T 1c2c7375029a4839bad10b59008830f9 P a7d3e2f7b2d8486fbd3c67a10f6f9447 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:36:17.259388 438 raft_consensus.cc:729] T 1c2c7375029a4839bad10b59008830f9 P a7d3e2f7b2d8486fbd3c67a10f6f9447 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7d3e2f7b2d8486fbd3c67a10f6f9447, State: Initialized, Role: FOLLOWER
I20260928 06:36:17.259387 439 raft_consensus.cc:374] T 1c2c7375029a4839bad10b59008830f9 P e0f3d400343544cc93a22d3c756e7838 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:36:17.259428 439 raft_consensus.cc:729] T 1c2c7375029a4839bad10b59008830f9 P e0f3d400343544cc93a22d3c756e7838 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0f3d400343544cc93a22d3c756e7838, State: Initialized, Role: FOLLOWER
I20260928 06:36:17.259555 438 consensus_queue.cc:260] T 1c2c7375029a4839bad10b59008830f9 P a7d3e2f7b2d8486fbd3c67a10f6f9447 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f0c1fd75e8472680dc5e3148a9c42b" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 33161 } } peers { permanent_uuid: "a7d3e2f7b2d8486fbd3c67a10f6f9447" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 34633 } } peers { permanent_uuid: "e0f3d400343544cc93a22d3c756e7838" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 45055 } }
I20260928 06:36:17.259569 439 consensus_queue.cc:260] T 1c2c7375029a4839bad10b59008830f9 P e0f3d400343544cc93a22d3c756e7838 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f0c1fd75e8472680dc5e3148a9c42b" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 33161 } } peers { permanent_uuid: "a7d3e2f7b2d8486fbd3c67a10f6f9447" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 34633 } } peers { permanent_uuid: "e0f3d400343544cc93a22d3c756e7838" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 45055 } }
I20260928 06:36:17.259873 439 ts_tablet_manager.cc:1428] T 1c2c7375029a4839bad10b59008830f9 P e0f3d400343544cc93a22d3c756e7838: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:36:17.259888 438 ts_tablet_manager.cc:1428] T 1c2c7375029a4839bad10b59008830f9 P a7d3e2f7b2d8486fbd3c67a10f6f9447: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260928 06:36:17.260123 32754 heartbeater.cc:503] Master 127.16.157.126:33113 was elected leader, sending a full tablet report...
I20260928 06:36:17.260143 32620 heartbeater.cc:503] Master 127.16.157.126:33113 was elected leader, sending a full tablet report...
I20260928 06:36:17.260245 437 raft_consensus.cc:348] T 1c2c7375029a4839bad10b59008830f9 P 17f0c1fd75e8472680dc5e3148a9c42b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f0c1fd75e8472680dc5e3148a9c42b" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 33161 } } peers { permanent_uuid: "a7d3e2f7b2d8486fbd3c67a10f6f9447" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 34633 } } peers { permanent_uuid: "e0f3d400343544cc93a22d3c756e7838" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 45055 } }
I20260928 06:36:17.260413 437 raft_consensus.cc:374] T 1c2c7375029a4839bad10b59008830f9 P 17f0c1fd75e8472680dc5e3148a9c42b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:36:17.260466 437 raft_consensus.cc:729] T 1c2c7375029a4839bad10b59008830f9 P 17f0c1fd75e8472680dc5e3148a9c42b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17f0c1fd75e8472680dc5e3148a9c42b, State: Initialized, Role: FOLLOWER
I20260928 06:36:17.260582 437 consensus_queue.cc:260] T 1c2c7375029a4839bad10b59008830f9 P 17f0c1fd75e8472680dc5e3148a9c42b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f0c1fd75e8472680dc5e3148a9c42b" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 33161 } } peers { permanent_uuid: "a7d3e2f7b2d8486fbd3c67a10f6f9447" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 34633 } } peers { permanent_uuid: "e0f3d400343544cc93a22d3c756e7838" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 45055 } }
I20260928 06:36:17.260797 437 ts_tablet_manager.cc:1428] T 1c2c7375029a4839bad10b59008830f9 P 17f0c1fd75e8472680dc5e3148a9c42b: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
I20260928 06:36:17.264070 420 heartbeater.cc:503] Master 127.16.157.126:33113 was elected leader, sending a full tablet report...
W20260928 06:36:17.310086 32755 tablet.cc:2369] T 1c2c7375029a4839bad10b59008830f9 P a7d3e2f7b2d8486fbd3c67a10f6f9447: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:36:17.403021 32621 tablet.cc:2369] T 1c2c7375029a4839bad10b59008830f9 P e0f3d400343544cc93a22d3c756e7838: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:36:17.466503 421 tablet.cc:2369] T 1c2c7375029a4839bad10b59008830f9 P 17f0c1fd75e8472680dc5e3148a9c42b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:36:17.467461 445 raft_consensus.cc:482] T 1c2c7375029a4839bad10b59008830f9 P 17f0c1fd75e8472680dc5e3148a9c42b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:36:17.467563 445 raft_consensus.cc:504] T 1c2c7375029a4839bad10b59008830f9 P 17f0c1fd75e8472680dc5e3148a9c42b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f0c1fd75e8472680dc5e3148a9c42b" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 33161 } } peers { permanent_uuid: "a7d3e2f7b2d8486fbd3c67a10f6f9447" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 34633 } } peers { permanent_uuid: "e0f3d400343544cc93a22d3c756e7838" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 45055 } }
I20260928 06:36:17.467958 445 leader_election.cc:288] T 1c2c7375029a4839bad10b59008830f9 P 17f0c1fd75e8472680dc5e3148a9c42b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a7d3e2f7b2d8486fbd3c67a10f6f9447 (127.16.157.66:34633), e0f3d400343544cc93a22d3c756e7838 (127.16.157.65:45055)
I20260928 06:36:17.470903 422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:17.468194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.66:34633 (local address 127.16.157.67:43851)
0928 06:36:17.468238 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:17.468250 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:17.468297 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:17.468510 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:17.468511 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:17.468513 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:36:17.468590 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:17.468592 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:17.469263 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:17.469265 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:17.470041 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:17.470043 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:17.470635 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:17.470636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:17.470712 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:17.470715 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:36:17.470787 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 06:36:17.470829 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260928 06:36:17.470896 436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:17.468243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:43851 (local address 127.16.157.66:34633)
0928 06:36:17.468306 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:17.468309 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:17.468325 (+ 16us) server_negotiation.cc:407] Connection header received
0928 06:36:17.468361 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:17.468362 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:17.468372 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:17.468495 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:36:17.468690 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:17.469183 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:17.470134 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:17.470530 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:17.470634 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:17.470643 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:17.470792 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 06:36:17.470829 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 06:36:17.471073 446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:17.468324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.65:45055 (local address 127.16.157.67:38953)
0928 06:36:17.468545 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:17.468553 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:17.468615 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:17.468779 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:17.468780 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:17.468782 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:36:17.468863 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:17.468865 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:17.469491 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:17.469492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:17.470279 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:17.470281 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:17.470819 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:17.470821 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:17.470896 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:17.470899 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:36:17.470958 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 06:36:17.470997 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":142,"threads_started":1}
I20260928 06:36:17.471069 435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:17.468380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:38953 (local address 127.16.157.65:45055)
0928 06:36:17.468447 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:17.468449 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:17.468611 (+ 162us) server_negotiation.cc:407] Connection header received
0928 06:36:17.468700 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:17.468701 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:17.468713 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:17.468774 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:36:17.468931 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:17.469403 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:17.470371 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:17.470728 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:17.470806 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:17.470813 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:17.470965 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 06:36:17.471003 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 06:36:17.471262 32708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c2c7375029a4839bad10b59008830f9" candidate_uuid: "17f0c1fd75e8472680dc5e3148a9c42b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7d3e2f7b2d8486fbd3c67a10f6f9447" is_pre_election: true
I20260928 06:36:17.471407 32708 raft_consensus.cc:2454] T 1c2c7375029a4839bad10b59008830f9 P a7d3e2f7b2d8486fbd3c67a10f6f9447 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17f0c1fd75e8472680dc5e3148a9c42b in term 0.
I20260928 06:36:17.471654 309 leader_election.cc:302] T 1c2c7375029a4839bad10b59008830f9 P 17f0c1fd75e8472680dc5e3148a9c42b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17f0c1fd75e8472680dc5e3148a9c42b, a7d3e2f7b2d8486fbd3c67a10f6f9447; no voters:
I20260928 06:36:17.471757 32574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c2c7375029a4839bad10b59008830f9" candidate_uuid: "17f0c1fd75e8472680dc5e3148a9c42b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0f3d400343544cc93a22d3c756e7838" is_pre_election: true
I20260928 06:36:17.471925 32574 raft_consensus.cc:2454] T 1c2c7375029a4839bad10b59008830f9 P e0f3d400343544cc93a22d3c756e7838 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17f0c1fd75e8472680dc5e3148a9c42b in term 0.
I20260928 06:36:17.472040 445 raft_consensus.cc:2781] T 1c2c7375029a4839bad10b59008830f9 P 17f0c1fd75e8472680dc5e3148a9c42b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:36:17.472110 445 raft_consensus.cc:482] T 1c2c7375029a4839bad10b59008830f9 P 17f0c1fd75e8472680dc5e3148a9c42b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:36:17.472146 445 raft_consensus.cc:3037] T 1c2c7375029a4839bad10b59008830f9 P 17f0c1fd75e8472680dc5e3148a9c42b [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:36:17.472853 445 raft_consensus.cc:504] T 1c2c7375029a4839bad10b59008830f9 P 17f0c1fd75e8472680dc5e3148a9c42b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f0c1fd75e8472680dc5e3148a9c42b" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 33161 } } peers { permanent_uuid: "a7d3e2f7b2d8486fbd3c67a10f6f9447" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 34633 } } peers { permanent_uuid: "e0f3d400343544cc93a22d3c756e7838" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 45055 } }
I20260928 06:36:17.472988 445 leader_election.cc:288] T 1c2c7375029a4839bad10b59008830f9 P 17f0c1fd75e8472680dc5e3148a9c42b [CANDIDATE]: Term 1 election: Requested vote from peers a7d3e2f7b2d8486fbd3c67a10f6f9447 (127.16.157.66:34633), e0f3d400343544cc93a22d3c756e7838 (127.16.157.65:45055)
I20260928 06:36:17.473150 32708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c2c7375029a4839bad10b59008830f9" candidate_uuid: "17f0c1fd75e8472680dc5e3148a9c42b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7d3e2f7b2d8486fbd3c67a10f6f9447"
I20260928 06:36:17.473233 32708 raft_consensus.cc:3037] T 1c2c7375029a4839bad10b59008830f9 P a7d3e2f7b2d8486fbd3c67a10f6f9447 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:36:17.473237 32574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c2c7375029a4839bad10b59008830f9" candidate_uuid: "17f0c1fd75e8472680dc5e3148a9c42b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0f3d400343544cc93a22d3c756e7838"
I20260928 06:36:17.473331 32574 raft_consensus.cc:3037] T 1c2c7375029a4839bad10b59008830f9 P e0f3d400343544cc93a22d3c756e7838 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:36:17.474118 32708 raft_consensus.cc:2454] T 1c2c7375029a4839bad10b59008830f9 P a7d3e2f7b2d8486fbd3c67a10f6f9447 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17f0c1fd75e8472680dc5e3148a9c42b in term 1.
I20260928 06:36:17.474185 32574 raft_consensus.cc:2454] T 1c2c7375029a4839bad10b59008830f9 P e0f3d400343544cc93a22d3c756e7838 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17f0c1fd75e8472680dc5e3148a9c42b in term 1.
I20260928 06:36:17.474318 309 leader_election.cc:302] T 1c2c7375029a4839bad10b59008830f9 P 17f0c1fd75e8472680dc5e3148a9c42b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17f0c1fd75e8472680dc5e3148a9c42b, a7d3e2f7b2d8486fbd3c67a10f6f9447; no voters:
I20260928 06:36:17.474454 445 raft_consensus.cc:2781] T 1c2c7375029a4839bad10b59008830f9 P 17f0c1fd75e8472680dc5e3148a9c42b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:36:17.474643 445 raft_consensus.cc:686] T 1c2c7375029a4839bad10b59008830f9 P 17f0c1fd75e8472680dc5e3148a9c42b [term 1 LEADER]: Becoming Leader. State: Replica: 17f0c1fd75e8472680dc5e3148a9c42b, State: Running, Role: LEADER
I20260928 06:36:17.474763 445 consensus_queue.cc:237] T 1c2c7375029a4839bad10b59008830f9 P 17f0c1fd75e8472680dc5e3148a9c42b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f0c1fd75e8472680dc5e3148a9c42b" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 33161 } } peers { permanent_uuid: "a7d3e2f7b2d8486fbd3c67a10f6f9447" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 34633 } } peers { permanent_uuid: "e0f3d400343544cc93a22d3c756e7838" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 45055 } }
I20260928 06:36:17.475584 32428 catalog_manager.cc:5495] T 1c2c7375029a4839bad10b59008830f9 P 17f0c1fd75e8472680dc5e3148a9c42b reported cstate change: term changed from 0 to 1, leader changed from <none> to 17f0c1fd75e8472680dc5e3148a9c42b (127.16.157.67). New cstate: current_term: 1 leader_uuid: "17f0c1fd75e8472680dc5e3148a9c42b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f0c1fd75e8472680dc5e3148a9c42b" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 33161 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a7d3e2f7b2d8486fbd3c67a10f6f9447" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 34633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0f3d400343544cc93a22d3c756e7838" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 45055 } health_report { overall_health: UNKNOWN } } }
I20260928 06:36:17.532404 434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:17.530084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59446 (local address 127.16.157.67:33161)
0928 06:36:17.530144 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:17.530146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:17.530172 (+ 26us) server_negotiation.cc:407] Connection header received
0928 06:36:17.530213 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:17.530214 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:17.530233 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:17.530287 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:36:17.530399 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:17.530874 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:17.531620 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:17.531867 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:17.531914 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:17.532053 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:36:17.532176 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:36:17.532235 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:17.532247 (+ 12us) server_negotiation.cc:299] Negotiation successful
0928 06:36:17.532268 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 06:36:17.534468 32708 raft_consensus.cc:1261] T 1c2c7375029a4839bad10b59008830f9 P a7d3e2f7b2d8486fbd3c67a10f6f9447 [term 1 FOLLOWER]: Refusing update from remote peer 17f0c1fd75e8472680dc5e3148a9c42b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:36:17.534468 32574 raft_consensus.cc:1261] T 1c2c7375029a4839bad10b59008830f9 P e0f3d400343544cc93a22d3c756e7838 [term 1 FOLLOWER]: Refusing update from remote peer 17f0c1fd75e8472680dc5e3148a9c42b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:36:17.534878 447 consensus_queue.cc:1035] T 1c2c7375029a4839bad10b59008830f9 P 17f0c1fd75e8472680dc5e3148a9c42b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7d3e2f7b2d8486fbd3c67a10f6f9447" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 34633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:36:17.535089 445 consensus_queue.cc:1035] T 1c2c7375029a4839bad10b59008830f9 P 17f0c1fd75e8472680dc5e3148a9c42b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0f3d400343544cc93a22d3c756e7838" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 45055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:36:17.538340 451 mvcc.cc:205] Tried to move back new op lower bound from 7334204938377797632 to 7334204938137063424. Current Snapshot: MvccSnapshot[applied={T|T < 7334204938377797632}]
I20260928 06:36:17.538488 452 mvcc.cc:205] Tried to move back new op lower bound from 7334204938377797632 to 7334204938137063424. Current Snapshot: MvccSnapshot[applied={T|T < 7334204938377797632}]
I20260928 06:36:17.539776 32429 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46440:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:36:17.539861 32429 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46440:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:36:17.540588 32429 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aae1e59bc3dd4b2c9599a522cbf5b565: Sending DeleteTablet for 3 replicas of tablet 1c2c7375029a4839bad10b59008830f9
I20260928 06:36:17.540967 354 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c2c7375029a4839bad10b59008830f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:36:17 UTC) from {username='kudu', principal='kudu/127.16.157.126@KRBTEST.COM'} at 127.0.0.1:59434
I20260928 06:36:17.541010 32688 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c2c7375029a4839bad10b59008830f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:36:17 UTC) from {username='kudu', principal='kudu/127.16.157.126@KRBTEST.COM'} at 127.0.0.1:42488
I20260928 06:36:17.541050 32554 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c2c7375029a4839bad10b59008830f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:36:17 UTC) from {username='kudu', principal='kudu/127.16.157.126@KRBTEST.COM'} at 127.0.0.1:45518
I20260928 06:36:17.541230 460 tablet_replica.cc:330] T 1c2c7375029a4839bad10b59008830f9 P a7d3e2f7b2d8486fbd3c67a10f6f9447: stopping tablet replica
I20260928 06:36:17.541281 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 32491
I20260928 06:36:17.541363 461 tablet_replica.cc:330] T 1c2c7375029a4839bad10b59008830f9 P e0f3d400343544cc93a22d3c756e7838: stopping tablet replica
I20260928 06:36:17.541488 461 raft_consensus.cc:2229] T 1c2c7375029a4839bad10b59008830f9 P e0f3d400343544cc93a22d3c756e7838 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:36:17.541664 461 raft_consensus.cc:2258] T 1c2c7375029a4839bad10b59008830f9 P e0f3d400343544cc93a22d3c756e7838 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:36:17.541342 460 raft_consensus.cc:2229] T 1c2c7375029a4839bad10b59008830f9 P a7d3e2f7b2d8486fbd3c67a10f6f9447 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:36:17.543365 460 raft_consensus.cc:2258] T 1c2c7375029a4839bad10b59008830f9 P a7d3e2f7b2d8486fbd3c67a10f6f9447 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:36:17.543819 460 ts_tablet_manager.cc:1905] T 1c2c7375029a4839bad10b59008830f9 P a7d3e2f7b2d8486fbd3c67a10f6f9447: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:36:17.545436 460 ts_tablet_manager.cc:1918] T 1c2c7375029a4839bad10b59008830f9 P a7d3e2f7b2d8486fbd3c67a10f6f9447: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:36:17.545516 460 log.cc:1196] T 1c2c7375029a4839bad10b59008830f9 P a7d3e2f7b2d8486fbd3c67a10f6f9447: Deleting WAL directory at /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-1/wal/wals/1c2c7375029a4839bad10b59008830f9
I20260928 06:36:17.545802 460 ts_tablet_manager.cc:1939] T 1c2c7375029a4839bad10b59008830f9 P a7d3e2f7b2d8486fbd3c67a10f6f9447: Deleting consensus metadata
I20260928 06:36:17.546274 32415 catalog_manager.cc:4841] TS a7d3e2f7b2d8486fbd3c67a10f6f9447 (127.16.157.66:34633): tablet 1c2c7375029a4839bad10b59008830f9 (table test-table [id=be71b3c386074100910cbb3fb4f024bc]) successfully deleted
W20260928 06:36:17.547793 32415 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.157.65:45055 (error 108)
I20260928 06:36:17.547956 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 32625
I20260928 06:36:17.549130 459 tablet_replica.cc:330] T 1c2c7375029a4839bad10b59008830f9 P 17f0c1fd75e8472680dc5e3148a9c42b: stopping tablet replica
I20260928 06:36:17.549247 459 raft_consensus.cc:2229] T 1c2c7375029a4839bad10b59008830f9 P 17f0c1fd75e8472680dc5e3148a9c42b [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:36:17.549355 459 raft_consensus.cc:2258] T 1c2c7375029a4839bad10b59008830f9 P 17f0c1fd75e8472680dc5e3148a9c42b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:36:17.550249 432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:17.550108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.65:45055 (local address 127.0.0.1:45520)
0928 06:36:17.550163 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:17.550206 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.65:45055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 06:36:17.550244 459 ts_tablet_manager.cc:1905] T 1c2c7375029a4839bad10b59008830f9 P 17f0c1fd75e8472680dc5e3148a9c42b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 06:36:17.550458 32415 catalog_manager.cc:4584] TS e0f3d400343544cc93a22d3c756e7838 (127.16.157.65:45055): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1c2c7375029a4839bad10b59008830f9: Network error: Client connection negotiation failed: client connection to 127.16.157.65:45055: connect: Connection refused (error 111)
I20260928 06:36:17.551945 459 ts_tablet_manager.cc:1918] T 1c2c7375029a4839bad10b59008830f9 P 17f0c1fd75e8472680dc5e3148a9c42b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:36:17.552026 459 log.cc:1196] T 1c2c7375029a4839bad10b59008830f9 P 17f0c1fd75e8472680dc5e3148a9c42b: Deleting WAL directory at /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-2/wal/wals/1c2c7375029a4839bad10b59008830f9
I20260928 06:36:17.552340 459 ts_tablet_manager.cc:1939] T 1c2c7375029a4839bad10b59008830f9 P 17f0c1fd75e8472680dc5e3148a9c42b: Deleting consensus metadata
I20260928 06:36:17.552790 32416 catalog_manager.cc:4841] TS 17f0c1fd75e8472680dc5e3148a9c42b (127.16.157.67:33161): tablet 1c2c7375029a4839bad10b59008830f9 (table test-table [id=be71b3c386074100910cbb3fb4f024bc]) successfully deleted
I20260928 06:36:17.554271 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 32759
I20260928 06:36:17.559881 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 32397
I20260928 06:36:37.565838 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:33113
--webserver_interface=127.16.157.126
--webserver_port=40755
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=kudu/127.16.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:43161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.157.126:33113
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:36:37.646037 463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:36:37.646353 463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:36:37.646391 463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:36:37.647905 463 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 06:36:37.647960 463 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 06:36:37.647982 463 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 06:36:37.648010 463 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:36:37.648027 463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:36:37.648046 463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:36:37.648063 463 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:36:37.648087 463 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:36:37.649729 463 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:43161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.16.157.126:33113
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=kudu/127.16.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:33113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.157.126
--webserver_port=40755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:36:37.650002 463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:36:37.650233 463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:36:37.659121 469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:37.659122 463 server_base.cc:1034] running on GCE node
W20260928 06:36:37.659217 470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:37.659416 472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:37.659660 463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:36:37.659976 463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:36:37.661147 463 hybrid_clock.cc:648] HybridClock initialized: now 1790577397661146 us; error 36 us; skew 500 ppm
Sep 28 06:36:37 dist-test-slave-rrj5 krb5kdc[32384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577397, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:36:37.663563 463 init.cc:377] Logged in from keytab as kudu/127.16.157.126@KRBTEST.COM (short username kudu)
I20260928 06:36:37.663986 463 webserver.cc:460] Webserver started at http://127.16.157.126:40755/ using document root <none> and password file <none>
I20260928 06:36:37.664220 463 fs_manager.cc:359] Metadata directory not provided
I20260928 06:36:37.664310 463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:36:37.665719 463 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:36:37.666381 479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:37.666563 463 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:36:37.666661 463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/master-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/master-0/wal
uuid: "aae1e59bc3dd4b2c9599a522cbf5b565"
format_stamp: "Formatted at 2026-09-28 06:36:16 on dist-test-slave-rrj5"
server_key: "135668f5271aa01d9c65290e12cab450"
server_key_iv: "e688892c5d0ed37454c2914e627fcf4c"
server_key_version: "encryptionkey@0"
I20260928 06:36:37.667114 463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:36:37.686784 463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:36:37.688552 463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:36:37.688714 463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:36:37.692790 463 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.126:33113
I20260928 06:36:37.692827 531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.126:33113 every 8 connection(s)
I20260928 06:36:37.693235 463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
I20260928 06:36:37.694782 532 sys_catalog.cc:263] Verifying existing consensus state
I20260928 06:36:37.695443 532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aae1e59bc3dd4b2c9599a522cbf5b565: Bootstrap starting.
I20260928 06:36:37.701332 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 463
I20260928 06:36:37.701685 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.65:45055
--local_ip_for_outbound_sockets=127.16.157.65
--tserver_master_addrs=127.16.157.126:33113
--webserver_port=35837
--webserver_interface=127.16.157.65
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab
--principal=kudu/127.16.157.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:43161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:36:37.704761 532 log.cc:824] T 00000000000000000000000000000000 P aae1e59bc3dd4b2c9599a522cbf5b565: Log is configured to *not* fsync() on all Append() calls
I20260928 06:36:37.706759 532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aae1e59bc3dd4b2c9599a522cbf5b565: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:36:37.707054 532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aae1e59bc3dd4b2c9599a522cbf5b565: Bootstrap complete.
I20260928 06:36:37.709304 532 raft_consensus.cc:348] T 00000000000000000000000000000000 P aae1e59bc3dd4b2c9599a522cbf5b565 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aae1e59bc3dd4b2c9599a522cbf5b565" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 33113 } }
I20260928 06:36:37.709509 532 raft_consensus.cc:729] T 00000000000000000000000000000000 P aae1e59bc3dd4b2c9599a522cbf5b565 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae1e59bc3dd4b2c9599a522cbf5b565, State: Initialized, Role: FOLLOWER
I20260928 06:36:37.709681 532 consensus_queue.cc:260] T 00000000000000000000000000000000 P aae1e59bc3dd4b2c9599a522cbf5b565 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aae1e59bc3dd4b2c9599a522cbf5b565" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 33113 } }
I20260928 06:36:37.709740 532 raft_consensus.cc:388] T 00000000000000000000000000000000 P aae1e59bc3dd4b2c9599a522cbf5b565 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:36:37.709800 532 raft_consensus.cc:482] T 00000000000000000000000000000000 P aae1e59bc3dd4b2c9599a522cbf5b565 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:36:37.709853 532 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aae1e59bc3dd4b2c9599a522cbf5b565 [term 1 FOLLOWER]: Advancing to term 2
I20260928 06:36:37.710806 532 raft_consensus.cc:504] T 00000000000000000000000000000000 P aae1e59bc3dd4b2c9599a522cbf5b565 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aae1e59bc3dd4b2c9599a522cbf5b565" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 33113 } }
I20260928 06:36:37.710888 532 leader_election.cc:302] T 00000000000000000000000000000000 P aae1e59bc3dd4b2c9599a522cbf5b565 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aae1e59bc3dd4b2c9599a522cbf5b565; no voters:
I20260928 06:36:37.711074 532 leader_election.cc:288] T 00000000000000000000000000000000 P aae1e59bc3dd4b2c9599a522cbf5b565 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 06:36:37.711108 536 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aae1e59bc3dd4b2c9599a522cbf5b565 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 06:36:37.711226 536 raft_consensus.cc:686] T 00000000000000000000000000000000 P aae1e59bc3dd4b2c9599a522cbf5b565 [term 2 LEADER]: Becoming Leader. State: Replica: aae1e59bc3dd4b2c9599a522cbf5b565, State: Running, Role: LEADER
I20260928 06:36:37.711325 536 consensus_queue.cc:237] T 00000000000000000000000000000000 P aae1e59bc3dd4b2c9599a522cbf5b565 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aae1e59bc3dd4b2c9599a522cbf5b565" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 33113 } }
I20260928 06:36:37.711370 532 sys_catalog.cc:564] T 00000000000000000000000000000000 P aae1e59bc3dd4b2c9599a522cbf5b565 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:36:37.712225 536 sys_catalog.cc:455] T 00000000000000000000000000000000 P aae1e59bc3dd4b2c9599a522cbf5b565 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "aae1e59bc3dd4b2c9599a522cbf5b565" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aae1e59bc3dd4b2c9599a522cbf5b565" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 33113 } } }
I20260928 06:36:37.712306 536 sys_catalog.cc:458] T 00000000000000000000000000000000 P aae1e59bc3dd4b2c9599a522cbf5b565 [sys.catalog]: This master's current role is: LEADER
I20260928 06:36:37.712450 536 sys_catalog.cc:455] T 00000000000000000000000000000000 P aae1e59bc3dd4b2c9599a522cbf5b565 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aae1e59bc3dd4b2c9599a522cbf5b565. Latest consensus state: current_term: 2 leader_uuid: "aae1e59bc3dd4b2c9599a522cbf5b565" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aae1e59bc3dd4b2c9599a522cbf5b565" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 33113 } } }
I20260928 06:36:37.712522 536 sys_catalog.cc:458] T 00000000000000000000000000000000 P aae1e59bc3dd4b2c9599a522cbf5b565 [sys.catalog]: This master's current role is: LEADER
I20260928 06:36:37.712597 546 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:36:37.713515 546 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:36:37.713646 546 catalog_manager.cc:1259] Loaded cluster ID: 40c2479ccc80479fa303215377f5c4eb
I20260928 06:36:37.713682 546 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:36:37.714841 546 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:36:37.715004 546 catalog_manager.cc:5879] T 00000000000000000000000000000000 P aae1e59bc3dd4b2c9599a522cbf5b565: Loaded TSK: 0
I20260928 06:36:37.725396 546 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aae1e59bc3dd4b2c9599a522cbf5b565: Generated new TSK 1
I20260928 06:36:37.725479 546 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 06:36:37.787806 534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:36:37.787998 534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:36:37.788020 534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:36:37.789104 534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:36:37.789144 534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:36:37.789170 534 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.65
I20260928 06:36:37.790588 534 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:43161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab
--principal=kudu/127.16.157.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.65:45055
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.157.65
--webserver_port=35837
--tserver_master_addrs=127.16.157.126:33113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.534
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.65
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:36:37.790809 534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:36:37.791047 534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:36:37.795950 559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:37.795986 560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:37.795991 562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:37.796154 534 server_base.cc:1034] running on GCE node
I20260928 06:36:37.796384 534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:36:37.796635 534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:36:37.797791 534 hybrid_clock.cc:648] HybridClock initialized: now 1790577397797774 us; error 32 us; skew 500 ppm
Sep 28 06:36:37 dist-test-slave-rrj5 krb5kdc[32384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577397, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:36:37.800226 534 init.cc:377] Logged in from keytab as kudu/127.16.157.65@KRBTEST.COM (short username kudu)
I20260928 06:36:37.800705 534 webserver.cc:460] Webserver started at http://127.16.157.65:35837/ using document root <none> and password file <none>
I20260928 06:36:37.800989 534 fs_manager.cc:359] Metadata directory not provided
I20260928 06:36:37.801043 534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:36:37.802508 534 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:36:37.803206 569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:37.803355 534 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:36:37.803457 534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-0/wal
uuid: "e0f3d400343544cc93a22d3c756e7838"
format_stamp: "Formatted at 2026-09-28 06:36:16 on dist-test-slave-rrj5"
server_key: "efcde22511bade5ec702202e732afb3e"
server_key_iv: "fbe53357b50bc140b9b6685798859930"
server_key_version: "encryptionkey@0"
I20260928 06:36:37.803786 534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:36:37.824265 534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:36:37.826385 534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:36:37.826545 534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:36:37.826807 534 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:36:37.827316 576 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 06:36:37.828714 534 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 06:36:37.828763 534 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 06:36:37.828794 534 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 06:36:37.829444 534 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 06:36:37.829489 534 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 06:36:37.829528 576 tablet_bootstrap.cc:492] T 1c2c7375029a4839bad10b59008830f9 P e0f3d400343544cc93a22d3c756e7838: Bootstrap starting.
I20260928 06:36:37.837335 534 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.65:45055
I20260928 06:36:37.837416 683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.65:45055 every 8 connection(s)
I20260928 06:36:37.837827 534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
I20260928 06:36:37.841149 576 log.cc:824] T 1c2c7375029a4839bad10b59008830f9 P e0f3d400343544cc93a22d3c756e7838: Log is configured to *not* fsync() on all Append() calls
I20260928 06:36:37.843781 576 tablet_bootstrap.cc:492] T 1c2c7375029a4839bad10b59008830f9 P e0f3d400343544cc93a22d3c756e7838: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:36:37.844179 576 tablet_bootstrap.cc:492] T 1c2c7375029a4839bad10b59008830f9 P e0f3d400343544cc93a22d3c756e7838: Bootstrap complete.
Sep 28 06:36:37 dist-test-slave-rrj5 krb5kdc[32384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577397, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.65@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:36:37.845373 576 ts_tablet_manager.cc:1397] T 1c2c7375029a4839bad10b59008830f9 P e0f3d400343544cc93a22d3c756e7838: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.012s
I20260928 06:36:37.846338 576 raft_consensus.cc:348] T 1c2c7375029a4839bad10b59008830f9 P e0f3d400343544cc93a22d3c756e7838 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f0c1fd75e8472680dc5e3148a9c42b" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 33161 } } peers { permanent_uuid: "a7d3e2f7b2d8486fbd3c67a10f6f9447" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 34633 } } peers { permanent_uuid: "e0f3d400343544cc93a22d3c756e7838" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 45055 } }
I20260928 06:36:37.846578 576 raft_consensus.cc:729] T 1c2c7375029a4839bad10b59008830f9 P e0f3d400343544cc93a22d3c756e7838 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0f3d400343544cc93a22d3c756e7838, State: Initialized, Role: FOLLOWER
I20260928 06:36:37.846750 576 consensus_queue.cc:260] T 1c2c7375029a4839bad10b59008830f9 P e0f3d400343544cc93a22d3c756e7838 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f0c1fd75e8472680dc5e3148a9c42b" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 33161 } } peers { permanent_uuid: "a7d3e2f7b2d8486fbd3c67a10f6f9447" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 34633 } } peers { permanent_uuid: "e0f3d400343544cc93a22d3c756e7838" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 45055 } }
I20260928 06:36:37.847074 576 ts_tablet_manager.cc:1428] T 1c2c7375029a4839bad10b59008830f9 P e0f3d400343544cc93a22d3c756e7838: Time spent starting tablet: real 0.002s user 0.003s sys 0.002s
I20260928 06:36:37.847357 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 534
I20260928 06:36:37.847612 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.66:34633
--local_ip_for_outbound_sockets=127.16.157.66
--tserver_master_addrs=127.16.157.126:33113
--webserver_port=34285
--webserver_interface=127.16.157.66
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab
--principal=kudu/127.16.157.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:43161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:36:37.849824 687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:37.838713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:50659 (local address 127.16.157.126:33113)
0928 06:36:37.838915 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:37.838923 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:37.838964 (+ 41us) server_negotiation.cc:407] Connection header received
0928 06:36:37.839571 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:37.839584 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:37.839628 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:37.839777 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:37.840405 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:37.841341 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:37.842917 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:37.843319 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:37.843420 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:37.845953 (+ 2533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:37.845957 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:37.845957 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:37.845968 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:37.847944 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:37.848364 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:37.848367 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:37.848367 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:37.848466 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:37.848789 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:37.848790 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:37.848791 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:37.849031 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:37.849103 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:37.849501 (+ 398us) server_negotiation.cc:299] Negotiation successful
0928 06:36:37.849668 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":104,"threads_started":1}
I20260928 06:36:37.849910 686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:37.838582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:33113 (local address 127.16.157.65:50659)
0928 06:36:37.838805 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:37.838848 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:37.839349 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:37.839817 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:37.839820 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:37.840093 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:37.840274 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:37.840279 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:37.841501 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:37.841502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:37.842813 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:37.842816 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:37.843454 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:37.843455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:37.843594 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:37.844172 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:37.844181 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:37.845843 (+ 1662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:36:37.848035 (+ 2192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:37.848041 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:37.848043 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:37.848266 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:37.848543 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:37.848544 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:37.848545 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:37.848700 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:37.849107 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:36:37.849114 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:36:37.849317 (+ 203us) client_negotiation.cc:769] Sending connection context
0928 06:36:37.849454 (+ 137us) client_negotiation.cc:240] Negotiation successful
0928 06:36:37.849664 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":127,"threads_started":1}
I20260928 06:36:37.850612 684 heartbeater.cc:347] Connected to a master server at 127.16.157.126:33113
I20260928 06:36:37.850694 684 heartbeater.cc:464] Registering TS with master...
I20260928 06:36:37.850884 684 heartbeater.cc:511] Master 127.16.157.126:33113 requested a full tablet report, sending...
I20260928 06:36:37.851619 496 ts_manager.cc:194] Registered new tserver with Master: e0f3d400343544cc93a22d3c756e7838 (127.16.157.65:45055)
I20260928 06:36:37.852681 496 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.157.65@KRBTEST.COM'} at 127.16.157.65:50659
I20260928 06:36:37.853729 684 heartbeater.cc:503] Master 127.16.157.126:33113 was elected leader, sending a full tablet report...
Sep 28 06:36:37 dist-test-slave-rrj5 krb5kdc[32384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577397, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.126@KRBTEST.COM for kudu/127.16.157.65@KRBTEST.COM
I20260928 06:36:37.861079 692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:37.852335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45140 (local address 127.16.157.65:45055)
0928 06:36:37.853010 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:37.853025 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:37.853036 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:36:37.853067 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:37.853069 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:37.853109 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:37.853200 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:37.853756 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:37.854947 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:37.855668 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:37.856172 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:37.856247 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:37.858534 (+ 2287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:37.858541 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:37.858542 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:37.858547 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:37.859935 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:37.860372 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:37.860374 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:37.860374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:37.860426 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:37.860610 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:37.860610 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:37.860610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:37.860726 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:37.860783 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:37.860949 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 06:36:37.861021 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":75,"threads_started":1}
I20260928 06:36:37.861135 691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:37.852260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.65:45055 (local address 127.0.0.1:45140)
0928 06:36:37.852412 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:37.852446 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:37.852507 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:37.853261 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:37.853267 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:37.853607 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:37.853665 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:37.853667 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:37.855031 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:37.855032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:37.855586 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:37.855587 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:37.856259 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:37.856260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:37.856316 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:37.856760 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:37.856764 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:37.858448 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:36:37.860013 (+ 1565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:37.860019 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:37.860021 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:37.860292 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:37.860490 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:37.860491 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:37.860492 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:37.860555 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:37.860784 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:36:37.860790 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:36:37.860840 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 06:36:37.860928 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 06:36:37.861037 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":85,"threads_started":1}
I20260928 06:36:37.861624 603 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c2c7375029a4839bad10b59008830f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:36:17 UTC) from {username='kudu', principal='kudu/127.16.157.126@KRBTEST.COM'} at 127.0.0.1:45140
I20260928 06:36:37.862187 693 tablet_replica.cc:330] stopping tablet replica
I20260928 06:36:37.862298 693 raft_consensus.cc:2229] T 1c2c7375029a4839bad10b59008830f9 P e0f3d400343544cc93a22d3c756e7838 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:36:37.862382 693 raft_consensus.cc:2258] T 1c2c7375029a4839bad10b59008830f9 P e0f3d400343544cc93a22d3c756e7838 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:36:37.862665 693 ts_tablet_manager.cc:1905] T 1c2c7375029a4839bad10b59008830f9 P e0f3d400343544cc93a22d3c756e7838: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:36:37.864827 693 ts_tablet_manager.cc:1918] T 1c2c7375029a4839bad10b59008830f9 P e0f3d400343544cc93a22d3c756e7838: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:36:37.864888 693 log.cc:1196] T 1c2c7375029a4839bad10b59008830f9 P e0f3d400343544cc93a22d3c756e7838: Deleting WAL directory at /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-0/wal/wals/1c2c7375029a4839bad10b59008830f9
I20260928 06:36:37.865142 693 ts_tablet_manager.cc:1939] T 1c2c7375029a4839bad10b59008830f9 P e0f3d400343544cc93a22d3c756e7838: Deleting consensus metadata
I20260928 06:36:37.865667 482 catalog_manager.cc:4841] TS e0f3d400343544cc93a22d3c756e7838 (127.16.157.65:45055): tablet 1c2c7375029a4839bad10b59008830f9 (table test-table [id=be71b3c386074100910cbb3fb4f024bc]) successfully deleted
W20260928 06:36:37.931751 690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:36:37.931982 690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:36:37.932022 690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:36:37.933216 690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:36:37.933279 690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:36:37.933322 690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.66
I20260928 06:36:37.934731 690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:43161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab
--principal=kudu/127.16.157.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.66:34633
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.157.66
--webserver_port=34285
--tserver_master_addrs=127.16.157.126:33113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.66
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:36:37.934981 690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:36:37.935205 690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:36:37.940372 701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:37.940389 699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:37.940435 698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:37.940943 690 server_base.cc:1034] running on GCE node
I20260928 06:36:37.941116 690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:36:37.941308 690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:36:37.942444 690 hybrid_clock.cc:648] HybridClock initialized: now 1790577397942432 us; error 26 us; skew 500 ppm
Sep 28 06:36:37 dist-test-slave-rrj5 krb5kdc[32384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577397, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:36:37.944864 690 init.cc:377] Logged in from keytab as kudu/127.16.157.66@KRBTEST.COM (short username kudu)
I20260928 06:36:37.945276 690 webserver.cc:460] Webserver started at http://127.16.157.66:34285/ using document root <none> and password file <none>
I20260928 06:36:37.945497 690 fs_manager.cc:359] Metadata directory not provided
I20260928 06:36:37.945592 690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:36:37.946898 690 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:36:37.947494 708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:37.947654 690 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:36:37.947764 690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-1/wal
uuid: "a7d3e2f7b2d8486fbd3c67a10f6f9447"
format_stamp: "Formatted at 2026-09-28 06:36:17 on dist-test-slave-rrj5"
server_key: "b32ea6555f8b48a6cdc0d85f0038d664"
server_key_iv: "e4234dfc0b751e95b47b1c06c45088e9"
server_key_version: "encryptionkey@0"
I20260928 06:36:37.948089 690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:36:37.975059 690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:36:37.977304 690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:36:37.977499 690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:36:37.977843 690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:36:37.978217 690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:36:37.978277 690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:37.978322 690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:36:37.978353 690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:37.984899 690 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.66:34633
I20260928 06:36:37.984961 821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.66:34633 every 8 connection(s)
I20260928 06:36:37.985309 690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
Sep 28 06:36:37 dist-test-slave-rrj5 krb5kdc[32384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577397, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.66@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:36:37.993695 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 690
I20260928 06:36:37.994010 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.67:33161
--local_ip_for_outbound_sockets=127.16.157.67
--tserver_master_addrs=127.16.157.126:33113
--webserver_port=36887
--webserver_interface=127.16.157.67
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab
--principal=kudu/127.16.157.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:43161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:36:37.994673 687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:37.986014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.66:53715 (local address 127.16.157.126:33113)
0928 06:36:37.986133 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:37.986135 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:37.986343 (+ 208us) server_negotiation.cc:407] Connection header received
0928 06:36:37.986858 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:37.986859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:37.986870 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:37.986924 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:37.987938 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:37.988426 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:37.989415 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:37.989680 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:37.989739 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:37.991835 (+ 2096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:37.991836 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:37.991837 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:37.991843 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:37.993175 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:37.993553 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:37.993554 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:37.993555 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:37.993639 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:37.993959 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:37.993960 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:37.993961 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:37.994103 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:37.994175 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:37.994524 (+ 349us) server_negotiation.cc:299] Negotiation successful
0928 06:36:37.994607 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 06:36:37.994948 824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:37.985985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:33113 (local address 127.16.157.66:53715)
0928 06:36:37.986212 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:37.986250 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:37.986746 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:37.986977 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:37.986980 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:37.987642 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:37.987831 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:37.987855 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:37.988512 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:37.988513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:37.989336 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:37.989338 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:37.989765 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:37.989766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:37.989871 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:37.990288 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:37.990295 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:37.991743 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:36:37.993253 (+ 1510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:37.993259 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:37.993261 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:37.993473 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:37.993713 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:37.993713 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:37.993714 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:37.993858 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:37.994208 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:36:37.994215 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:36:37.994372 (+ 157us) client_negotiation.cc:769] Sending connection context
0928 06:36:37.994491 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 06:36:37.994665 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":135,"threads_started":1}
I20260928 06:36:37.995528 822 heartbeater.cc:347] Connected to a master server at 127.16.157.126:33113
I20260928 06:36:37.995656 822 heartbeater.cc:464] Registering TS with master...
I20260928 06:36:37.995846 822 heartbeater.cc:511] Master 127.16.157.126:33113 requested a full tablet report, sending...
I20260928 06:36:37.996280 496 ts_manager.cc:194] Registered new tserver with Master: a7d3e2f7b2d8486fbd3c67a10f6f9447 (127.16.157.66:34633)
I20260928 06:36:37.996778 496 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.157.66@KRBTEST.COM'} at 127.16.157.66:53715
W20260928 06:36:38.077417 825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:36:38.077713 825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:36:38.077741 825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:36:38.078933 825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:36:38.078969 825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:36:38.078995 825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.67
I20260928 06:36:38.080519 825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:43161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab
--principal=kudu/127.16.157.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.67:33161
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.157.67
--webserver_port=36887
--tserver_master_addrs=127.16.157.126:33113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.67
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:36:38.080715 825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:36:38.080896 825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:36:38.086367 831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:38.086390 833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:38.086401 830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:38.086660 825 server_base.cc:1034] running on GCE node
I20260928 06:36:38.086848 825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:36:38.087046 825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:36:38.088196 825 hybrid_clock.cc:648] HybridClock initialized: now 1790577398088176 us; error 35 us; skew 500 ppm
Sep 28 06:36:38 dist-test-slave-rrj5 krb5kdc[32384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577398, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:36:38.090828 825 init.cc:377] Logged in from keytab as kudu/127.16.157.67@KRBTEST.COM (short username kudu)
I20260928 06:36:38.091343 825 webserver.cc:460] Webserver started at http://127.16.157.67:36887/ using document root <none> and password file <none>
I20260928 06:36:38.091593 825 fs_manager.cc:359] Metadata directory not provided
I20260928 06:36:38.091673 825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:36:38.093154 825 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 06:36:38.093775 840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:38.093966 825 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:36:38.094036 825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-2/wal
uuid: "17f0c1fd75e8472680dc5e3148a9c42b"
format_stamp: "Formatted at 2026-09-28 06:36:17 on dist-test-slave-rrj5"
server_key: "8a201b81bfc149b840eba2129582b3e7"
server_key_iv: "42b053b2b13010f5bfed8243f82a9c2b"
server_key_version: "encryptionkey@0"
I20260928 06:36:38.094388 825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:36:38.114137 825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:36:38.116217 825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:36:38.116374 825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:36:38.116638 825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:36:38.116984 825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:36:38.117040 825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:38.117076 825 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:36:38.117126 825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:38.123282 825 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.67:33161
I20260928 06:36:38.123342 953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.67:33161 every 8 connection(s)
I20260928 06:36:38.123721 825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
Sep 28 06:36:38 dist-test-slave-rrj5 krb5kdc[32384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577398, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.67@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:36:38.130038 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 825
I20260928 06:36:38.133448 687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:38.124489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:40587 (local address 127.16.157.126:33113)
0928 06:36:38.124577 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:38.124580 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:38.124807 (+ 227us) server_negotiation.cc:407] Connection header received
0928 06:36:38.125294 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:38.125295 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:38.125305 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:38.125359 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:38.126318 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:38.126762 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:38.127770 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:38.127959 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:38.128024 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:38.130307 (+ 2283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:38.130309 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:38.130309 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:38.130317 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:38.131960 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:38.132449 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:38.132451 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:38.132452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:38.132519 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:38.132768 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:38.132769 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:38.132769 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:38.132903 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:38.132972 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:38.133265 (+ 293us) server_negotiation.cc:299] Negotiation successful
0928 06:36:38.133358 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 06:36:38.133647 956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:38.124459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:33113 (local address 127.16.157.67:40587)
0928 06:36:38.124661 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:38.124697 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:38.125203 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:38.125411 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:38.125414 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:38.126056 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:38.126240 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:38.126245 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:38.126866 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:38.126866 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:38.127691 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:38.127693 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:38.128047 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:38.128048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:38.128148 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:38.128689 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:38.128697 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:38.130231 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:36:38.132045 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:38.132052 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:38.132055 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:38.132344 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:38.132582 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:38.132583 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:38.132583 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:38.132699 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:38.132971 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:36:38.132976 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:36:38.133091 (+ 115us) client_negotiation.cc:769] Sending connection context
0928 06:36:38.133212 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 06:36:38.133402 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":114,"threads_started":1}
I20260928 06:36:38.134271 954 heartbeater.cc:347] Connected to a master server at 127.16.157.126:33113
I20260928 06:36:38.134397 954 heartbeater.cc:464] Registering TS with master...
I20260928 06:36:38.134580 954 heartbeater.cc:511] Master 127.16.157.126:33113 requested a full tablet report, sending...
I20260928 06:36:38.135004 496 ts_manager.cc:194] Registered new tserver with Master: 17f0c1fd75e8472680dc5e3148a9c42b (127.16.157.67:33161)
I20260928 06:36:38.135383 496 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.157.67@KRBTEST.COM'} at 127.16.157.67:40587
I20260928 06:36:38.137751 957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:38.130456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51906 (local address 127.16.157.126:33113)
0928 06:36:38.130563 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:38.130565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:38.130677 (+ 112us) server_negotiation.cc:407] Connection header received
0928 06:36:38.130736 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:38.130737 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:38.130750 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:38.130820 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:38.131681 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:38.132352 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:38.133383 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:38.133724 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:38.133797 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:38.135164 (+ 1367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:38.135165 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:38.135165 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:38.135173 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:38.136640 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:38.137004 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:38.137005 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:38.137005 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:38.137062 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:38.137260 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:38.137261 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:38.137261 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:38.137363 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:38.137486 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:38.137512 (+ 26us) server_negotiation.cc:299] Negotiation successful
0928 06:36:38.137642 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":64,"threads_started":1}
I20260928 06:36:38.138511 17013 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:36:38.141029 957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:38.138916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51916 (local address 127.16.157.126:33113)
0928 06:36:38.138961 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:38.138963 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:38.139035 (+ 72us) server_negotiation.cc:407] Connection header received
0928 06:36:38.139092 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:38.139092 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:38.139124 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:38.139189 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:36:38.139269 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:38.139727 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:38.140445 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:38.140652 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:38.140716 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:38.140826 (+ 110us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:36:38.140865 (+ 39us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 06:36:38.140986 (+ 121us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51916: token expired
Metrics: {"server-negotiator.queue_time_us":26}
W20260928 06:36:38.141108 957 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51916: token expired
W20260928 06:36:38.141227 424 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.16.157.126:33113)
I20260928 06:36:38.146914 957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:38.141482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51920 (local address 127.16.157.126:33113)
0928 06:36:38.141516 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:38.141518 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:38.141526 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:36:38.141552 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:38.141553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:38.141562 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:38.141671 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:38.142164 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:38.142573 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:38.143192 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:38.143357 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:38.143409 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:38.144444 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:38.144445 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:38.144445 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:38.144451 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:38.146026 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:38.146378 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:38.146379 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:38.146379 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:38.146428 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:38.146575 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:38.146575 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:38.146576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:38.146658 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:38.146724 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:38.146805 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 06:36:38.146861 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 06:36:38.148003 496 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51920:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:36:38.149633 496 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:36:38.151845 603 tablet_service.cc:1467] Processing CreateTablet for tablet 6d4a8e680e9a494ebb52332d9cd47deb (DEFAULT_TABLE table=test-table [id=31746fb73da74f7cb35f6fbb218364bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:36:38.152055 603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d4a8e680e9a494ebb52332d9cd47deb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:36:38.153383 576 tablet_bootstrap.cc:492] T 6d4a8e680e9a494ebb52332d9cd47deb P e0f3d400343544cc93a22d3c756e7838: Bootstrap starting.
I20260928 06:36:38.153739 576 tablet_bootstrap.cc:654] T 6d4a8e680e9a494ebb52332d9cd47deb P e0f3d400343544cc93a22d3c756e7838: Neither blocks nor log segments found. Creating new log.
I20260928 06:36:38.154371 576 tablet_bootstrap.cc:492] T 6d4a8e680e9a494ebb52332d9cd47deb P e0f3d400343544cc93a22d3c756e7838: No bootstrap required, opened a new log
I20260928 06:36:38.154423 576 ts_tablet_manager.cc:1397] T 6d4a8e680e9a494ebb52332d9cd47deb P e0f3d400343544cc93a22d3c756e7838: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:36:38.154770 576 raft_consensus.cc:348] T 6d4a8e680e9a494ebb52332d9cd47deb P e0f3d400343544cc93a22d3c756e7838 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d3e2f7b2d8486fbd3c67a10f6f9447" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 34633 } } peers { permanent_uuid: "17f0c1fd75e8472680dc5e3148a9c42b" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 33161 } } peers { permanent_uuid: "e0f3d400343544cc93a22d3c756e7838" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 45055 } }
I20260928 06:36:38.154851 576 raft_consensus.cc:374] T 6d4a8e680e9a494ebb52332d9cd47deb P e0f3d400343544cc93a22d3c756e7838 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:36:38.154902 576 raft_consensus.cc:729] T 6d4a8e680e9a494ebb52332d9cd47deb P e0f3d400343544cc93a22d3c756e7838 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0f3d400343544cc93a22d3c756e7838, State: Initialized, Role: FOLLOWER
I20260928 06:36:38.154955 576 consensus_queue.cc:260] T 6d4a8e680e9a494ebb52332d9cd47deb P e0f3d400343544cc93a22d3c756e7838 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d3e2f7b2d8486fbd3c67a10f6f9447" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 34633 } } peers { permanent_uuid: "17f0c1fd75e8472680dc5e3148a9c42b" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 33161 } } peers { permanent_uuid: "e0f3d400343544cc93a22d3c756e7838" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 45055 } }
I20260928 06:36:38.155064 576 ts_tablet_manager.cc:1428] T 6d4a8e680e9a494ebb52332d9cd47deb P e0f3d400343544cc93a22d3c756e7838: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:36:38.156219 691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:38.151479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.66:34633 (local address 127.0.0.1:54358)
0928 06:36:38.151530 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:38.151542 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:38.151576 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:38.152246 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:38.152246 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:38.152248 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:36:38.152292 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:38.152294 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:38.153668 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:38.153668 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:38.154964 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:38.154967 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:38.155895 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:38.155896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:38.156000 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:38.156021 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 06:36:38.156082 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 06:36:38.156138 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":70}
I20260928 06:36:38.156231 962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:38.151883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54358 (local address 127.16.157.66:34633)
0928 06:36:38.152053 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:38.152057 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:38.152068 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:36:38.152113 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:38.152115 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:38.152131 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:38.152365 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:36:38.152455 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:38.153539 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:38.155056 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:38.155822 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:38.155885 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:38.155941 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:38.156102 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 06:36:38.156142 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":80,"threads_started":1}
I20260928 06:36:38.156486 960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:38.151533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.67:33161 (local address 127.0.0.1:34162)
0928 06:36:38.152068 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:38.152078 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:38.152147 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:38.152367 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:38.152368 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:38.152370 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:36:38.152412 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:38.152413 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:38.153641 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:38.153642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:38.155298 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:38.155301 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:38.156236 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:38.156237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:38.156298 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:38.156302 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:36:38.156357 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 06:36:38.156414 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"mutex_wait_us":661,"thread_start_us":112,"threads_started":1}
I20260928 06:36:38.156504 961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:38.151652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34162 (local address 127.16.157.67:33161)
0928 06:36:38.151843 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:38.151847 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:38.152211 (+ 364us) server_negotiation.cc:407] Connection header received
0928 06:36:38.152252 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:38.152254 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:38.152268 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:38.152346 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:36:38.152464 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:38.153518 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:38.155371 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:38.156139 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:38.156234 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:38.156287 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:38.156371 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 06:36:38.156427 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":97,"threads_started":1}
I20260928 06:36:38.157202 888 tablet_service.cc:1467] Processing CreateTablet for tablet 6d4a8e680e9a494ebb52332d9cd47deb (DEFAULT_TABLE table=test-table [id=31746fb73da74f7cb35f6fbb218364bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:36:38.157207 756 tablet_service.cc:1467] Processing CreateTablet for tablet 6d4a8e680e9a494ebb52332d9cd47deb (DEFAULT_TABLE table=test-table [id=31746fb73da74f7cb35f6fbb218364bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:36:38.157553 888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d4a8e680e9a494ebb52332d9cd47deb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:36:38.157553 756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d4a8e680e9a494ebb52332d9cd47deb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:36:38.161219 964 tablet_bootstrap.cc:492] T 6d4a8e680e9a494ebb52332d9cd47deb P a7d3e2f7b2d8486fbd3c67a10f6f9447: Bootstrap starting.
I20260928 06:36:38.161926 964 tablet_bootstrap.cc:654] T 6d4a8e680e9a494ebb52332d9cd47deb P a7d3e2f7b2d8486fbd3c67a10f6f9447: Neither blocks nor log segments found. Creating new log.
I20260928 06:36:38.162388 964 log.cc:824] T 6d4a8e680e9a494ebb52332d9cd47deb P a7d3e2f7b2d8486fbd3c67a10f6f9447: Log is configured to *not* fsync() on all Append() calls
I20260928 06:36:38.162675 965 tablet_bootstrap.cc:492] T 6d4a8e680e9a494ebb52332d9cd47deb P 17f0c1fd75e8472680dc5e3148a9c42b: Bootstrap starting.
I20260928 06:36:38.163072 964 tablet_bootstrap.cc:492] T 6d4a8e680e9a494ebb52332d9cd47deb P a7d3e2f7b2d8486fbd3c67a10f6f9447: No bootstrap required, opened a new log
I20260928 06:36:38.163157 964 ts_tablet_manager.cc:1397] T 6d4a8e680e9a494ebb52332d9cd47deb P a7d3e2f7b2d8486fbd3c67a10f6f9447: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 06:36:38.163218 965 tablet_bootstrap.cc:654] T 6d4a8e680e9a494ebb52332d9cd47deb P 17f0c1fd75e8472680dc5e3148a9c42b: Neither blocks nor log segments found. Creating new log.
I20260928 06:36:38.163713 965 log.cc:824] T 6d4a8e680e9a494ebb52332d9cd47deb P 17f0c1fd75e8472680dc5e3148a9c42b: Log is configured to *not* fsync() on all Append() calls
I20260928 06:36:38.164425 965 tablet_bootstrap.cc:492] T 6d4a8e680e9a494ebb52332d9cd47deb P 17f0c1fd75e8472680dc5e3148a9c42b: No bootstrap required, opened a new log
I20260928 06:36:38.164480 965 ts_tablet_manager.cc:1397] T 6d4a8e680e9a494ebb52332d9cd47deb P 17f0c1fd75e8472680dc5e3148a9c42b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:36:38.164685 964 raft_consensus.cc:348] T 6d4a8e680e9a494ebb52332d9cd47deb P a7d3e2f7b2d8486fbd3c67a10f6f9447 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d3e2f7b2d8486fbd3c67a10f6f9447" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 34633 } } peers { permanent_uuid: "17f0c1fd75e8472680dc5e3148a9c42b" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 33161 } } peers { permanent_uuid: "e0f3d400343544cc93a22d3c756e7838" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 45055 } }
I20260928 06:36:38.164928 964 raft_consensus.cc:374] T 6d4a8e680e9a494ebb52332d9cd47deb P a7d3e2f7b2d8486fbd3c67a10f6f9447 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:36:38.165063 964 raft_consensus.cc:729] T 6d4a8e680e9a494ebb52332d9cd47deb P a7d3e2f7b2d8486fbd3c67a10f6f9447 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7d3e2f7b2d8486fbd3c67a10f6f9447, State: Initialized, Role: FOLLOWER
I20260928 06:36:38.165226 964 consensus_queue.cc:260] T 6d4a8e680e9a494ebb52332d9cd47deb P a7d3e2f7b2d8486fbd3c67a10f6f9447 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d3e2f7b2d8486fbd3c67a10f6f9447" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 34633 } } peers { permanent_uuid: "17f0c1fd75e8472680dc5e3148a9c42b" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 33161 } } peers { permanent_uuid: "e0f3d400343544cc93a22d3c756e7838" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 45055 } }
I20260928 06:36:38.165534 964 ts_tablet_manager.cc:1428] T 6d4a8e680e9a494ebb52332d9cd47deb P a7d3e2f7b2d8486fbd3c67a10f6f9447: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 06:36:38.165555 822 heartbeater.cc:503] Master 127.16.157.126:33113 was elected leader, sending a full tablet report...
I20260928 06:36:38.166719 965 raft_consensus.cc:348] T 6d4a8e680e9a494ebb52332d9cd47deb P 17f0c1fd75e8472680dc5e3148a9c42b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d3e2f7b2d8486fbd3c67a10f6f9447" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 34633 } } peers { permanent_uuid: "17f0c1fd75e8472680dc5e3148a9c42b" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 33161 } } peers { permanent_uuid: "e0f3d400343544cc93a22d3c756e7838" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 45055 } }
I20260928 06:36:38.166888 965 raft_consensus.cc:374] T 6d4a8e680e9a494ebb52332d9cd47deb P 17f0c1fd75e8472680dc5e3148a9c42b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:36:38.166914 965 raft_consensus.cc:729] T 6d4a8e680e9a494ebb52332d9cd47deb P 17f0c1fd75e8472680dc5e3148a9c42b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17f0c1fd75e8472680dc5e3148a9c42b, State: Initialized, Role: FOLLOWER
I20260928 06:36:38.166991 965 consensus_queue.cc:260] T 6d4a8e680e9a494ebb52332d9cd47deb P 17f0c1fd75e8472680dc5e3148a9c42b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d3e2f7b2d8486fbd3c67a10f6f9447" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 34633 } } peers { permanent_uuid: "17f0c1fd75e8472680dc5e3148a9c42b" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 33161 } } peers { permanent_uuid: "e0f3d400343544cc93a22d3c756e7838" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 45055 } }
I20260928 06:36:38.167196 965 ts_tablet_manager.cc:1428] T 6d4a8e680e9a494ebb52332d9cd47deb P 17f0c1fd75e8472680dc5e3148a9c42b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:36:38.167268 954 heartbeater.cc:503] Master 127.16.157.126:33113 was elected leader, sending a full tablet report...
W20260928 06:36:38.235997 823 tablet.cc:2369] T 6d4a8e680e9a494ebb52332d9cd47deb P a7d3e2f7b2d8486fbd3c67a10f6f9447: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:36:38.338621 685 tablet.cc:2369] T 6d4a8e680e9a494ebb52332d9cd47deb P e0f3d400343544cc93a22d3c756e7838: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:36:38.374302 955 tablet.cc:2369] T 6d4a8e680e9a494ebb52332d9cd47deb P 17f0c1fd75e8472680dc5e3148a9c42b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:36:38.436961 689 raft_consensus.cc:482] T 6d4a8e680e9a494ebb52332d9cd47deb P e0f3d400343544cc93a22d3c756e7838 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:36:38.437134 689 raft_consensus.cc:504] T 6d4a8e680e9a494ebb52332d9cd47deb P e0f3d400343544cc93a22d3c756e7838 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d3e2f7b2d8486fbd3c67a10f6f9447" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 34633 } } peers { permanent_uuid: "17f0c1fd75e8472680dc5e3148a9c42b" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 33161 } } peers { permanent_uuid: "e0f3d400343544cc93a22d3c756e7838" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 45055 } }
I20260928 06:36:38.437525 689 leader_election.cc:288] T 6d4a8e680e9a494ebb52332d9cd47deb P e0f3d400343544cc93a22d3c756e7838 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a7d3e2f7b2d8486fbd3c67a10f6f9447 (127.16.157.66:34633), 17f0c1fd75e8472680dc5e3148a9c42b (127.16.157.67:33161)
I20260928 06:36:38.440553 970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:38.437703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.66:34633 (local address 127.16.157.65:56261)
0928 06:36:38.437832 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:38.437845 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:38.437926 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:38.438120 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:38.438121 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:38.438123 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:36:38.438191 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:38.438193 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:38.438858 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:38.438859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:38.439641 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:38.439642 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:38.440264 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:38.440265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:38.440328 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:38.440354 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 06:36:38.440420 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 06:36:38.440466 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":69,"threads_started":1}
I20260928 06:36:38.440574 962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:38.437774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:56261 (local address 127.16.157.66:34633)
0928 06:36:38.437850 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:38.437852 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:38.437898 (+ 46us) server_negotiation.cc:407] Connection header received
0928 06:36:38.438035 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:38.438036 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:38.438047 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:38.438092 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:36:38.438270 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:38.438768 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:38.439733 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:38.440182 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:38.440237 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:38.440244 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:38.440452 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 06:36:38.440495 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 06:36:38.440729 961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:38.437929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:38119 (local address 127.16.157.67:33161)
0928 06:36:38.438003 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:38.438005 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:38.438013 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:36:38.438041 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:38.438042 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:38.438051 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:38.438128 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:36:38.438329 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:38.438788 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:38.439924 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:38.440388 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:38.440454 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:38.440461 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:38.440624 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 06:36:38.440670 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 06:36:38.440887 971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:38.437743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.67:33161 (local address 127.16.157.65:38119)
0928 06:36:38.437862 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:38.437876 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:38.437935 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:38.438151 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:38.438152 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:38.438154 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:36:38.438245 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:38.438247 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:38.438900 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:38.438901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:38.439645 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:38.439646 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:38.440476 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:38.440477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:38.440536 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:38.440540 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:36:38.440599 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 06:36:38.440800 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"mutex_wait_us":14,"thread_start_us":74,"threads_started":1}
I20260928 06:36:38.441107 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d4a8e680e9a494ebb52332d9cd47deb" candidate_uuid: "e0f3d400343544cc93a22d3c756e7838" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7d3e2f7b2d8486fbd3c67a10f6f9447" is_pre_election: true
I20260928 06:36:38.441286 776 raft_consensus.cc:2454] T 6d4a8e680e9a494ebb52332d9cd47deb P a7d3e2f7b2d8486fbd3c67a10f6f9447 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0f3d400343544cc93a22d3c756e7838 in term 0.
I20260928 06:36:38.441259 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d4a8e680e9a494ebb52332d9cd47deb" candidate_uuid: "e0f3d400343544cc93a22d3c756e7838" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17f0c1fd75e8472680dc5e3148a9c42b" is_pre_election: true
I20260928 06:36:38.441421 908 raft_consensus.cc:2454] T 6d4a8e680e9a494ebb52332d9cd47deb P 17f0c1fd75e8472680dc5e3148a9c42b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0f3d400343544cc93a22d3c756e7838 in term 0.
I20260928 06:36:38.441520 572 leader_election.cc:302] T 6d4a8e680e9a494ebb52332d9cd47deb P e0f3d400343544cc93a22d3c756e7838 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7d3e2f7b2d8486fbd3c67a10f6f9447, e0f3d400343544cc93a22d3c756e7838; no voters:
I20260928 06:36:38.441696 689 raft_consensus.cc:2781] T 6d4a8e680e9a494ebb52332d9cd47deb P e0f3d400343544cc93a22d3c756e7838 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:36:38.441764 689 raft_consensus.cc:482] T 6d4a8e680e9a494ebb52332d9cd47deb P e0f3d400343544cc93a22d3c756e7838 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:36:38.441782 689 raft_consensus.cc:3037] T 6d4a8e680e9a494ebb52332d9cd47deb P e0f3d400343544cc93a22d3c756e7838 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:36:38.442422 689 raft_consensus.cc:504] T 6d4a8e680e9a494ebb52332d9cd47deb P e0f3d400343544cc93a22d3c756e7838 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d3e2f7b2d8486fbd3c67a10f6f9447" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 34633 } } peers { permanent_uuid: "17f0c1fd75e8472680dc5e3148a9c42b" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 33161 } } peers { permanent_uuid: "e0f3d400343544cc93a22d3c756e7838" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 45055 } }
I20260928 06:36:38.442567 689 leader_election.cc:288] T 6d4a8e680e9a494ebb52332d9cd47deb P e0f3d400343544cc93a22d3c756e7838 [CANDIDATE]: Term 1 election: Requested vote from peers a7d3e2f7b2d8486fbd3c67a10f6f9447 (127.16.157.66:34633), 17f0c1fd75e8472680dc5e3148a9c42b (127.16.157.67:33161)
I20260928 06:36:38.442754 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d4a8e680e9a494ebb52332d9cd47deb" candidate_uuid: "e0f3d400343544cc93a22d3c756e7838" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17f0c1fd75e8472680dc5e3148a9c42b"
I20260928 06:36:38.442742 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d4a8e680e9a494ebb52332d9cd47deb" candidate_uuid: "e0f3d400343544cc93a22d3c756e7838" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7d3e2f7b2d8486fbd3c67a10f6f9447"
I20260928 06:36:38.442823 908 raft_consensus.cc:3037] T 6d4a8e680e9a494ebb52332d9cd47deb P 17f0c1fd75e8472680dc5e3148a9c42b [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:36:38.442850 776 raft_consensus.cc:3037] T 6d4a8e680e9a494ebb52332d9cd47deb P a7d3e2f7b2d8486fbd3c67a10f6f9447 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:36:38.443513 908 raft_consensus.cc:2454] T 6d4a8e680e9a494ebb52332d9cd47deb P 17f0c1fd75e8472680dc5e3148a9c42b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0f3d400343544cc93a22d3c756e7838 in term 1.
I20260928 06:36:38.443660 776 raft_consensus.cc:2454] T 6d4a8e680e9a494ebb52332d9cd47deb P a7d3e2f7b2d8486fbd3c67a10f6f9447 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0f3d400343544cc93a22d3c756e7838 in term 1.
I20260928 06:36:38.443684 573 leader_election.cc:302] T 6d4a8e680e9a494ebb52332d9cd47deb P e0f3d400343544cc93a22d3c756e7838 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17f0c1fd75e8472680dc5e3148a9c42b, e0f3d400343544cc93a22d3c756e7838; no voters:
I20260928 06:36:38.443760 689 raft_consensus.cc:2781] T 6d4a8e680e9a494ebb52332d9cd47deb P e0f3d400343544cc93a22d3c756e7838 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:36:38.443867 689 raft_consensus.cc:686] T 6d4a8e680e9a494ebb52332d9cd47deb P e0f3d400343544cc93a22d3c756e7838 [term 1 LEADER]: Becoming Leader. State: Replica: e0f3d400343544cc93a22d3c756e7838, State: Running, Role: LEADER
I20260928 06:36:38.443940 689 consensus_queue.cc:237] T 6d4a8e680e9a494ebb52332d9cd47deb P e0f3d400343544cc93a22d3c756e7838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d3e2f7b2d8486fbd3c67a10f6f9447" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 34633 } } peers { permanent_uuid: "17f0c1fd75e8472680dc5e3148a9c42b" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 33161 } } peers { permanent_uuid: "e0f3d400343544cc93a22d3c756e7838" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 45055 } }
I20260928 06:36:38.444748 496 catalog_manager.cc:5495] T 6d4a8e680e9a494ebb52332d9cd47deb P e0f3d400343544cc93a22d3c756e7838 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0f3d400343544cc93a22d3c756e7838 (127.16.157.65). New cstate: current_term: 1 leader_uuid: "e0f3d400343544cc93a22d3c756e7838" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d3e2f7b2d8486fbd3c67a10f6f9447" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 34633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17f0c1fd75e8472680dc5e3148a9c42b" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 33161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0f3d400343544cc93a22d3c756e7838" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 45055 } health_report { overall_health: HEALTHY } } }
I20260928 06:36:38.510178 975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:38.507789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52144 (local address 127.16.157.65:45055)
0928 06:36:38.507969 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:38.507971 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:38.507980 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:36:38.508039 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:38.508040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:38.508058 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:38.508128 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:36:38.508251 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:38.508723 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:38.509373 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:38.509560 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:38.509662 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:38.509809 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:36:38.509954 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:36:38.510008 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:38.510043 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 06:36:38.510059 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":95,"threads_started":1}
I20260928 06:36:38.511772 908 raft_consensus.cc:1261] T 6d4a8e680e9a494ebb52332d9cd47deb P 17f0c1fd75e8472680dc5e3148a9c42b [term 1 FOLLOWER]: Refusing update from remote peer e0f3d400343544cc93a22d3c756e7838: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:36:38.511830 776 raft_consensus.cc:1261] T 6d4a8e680e9a494ebb52332d9cd47deb P a7d3e2f7b2d8486fbd3c67a10f6f9447 [term 1 FOLLOWER]: Refusing update from remote peer e0f3d400343544cc93a22d3c756e7838: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:36:38.512151 972 consensus_queue.cc:1035] T 6d4a8e680e9a494ebb52332d9cd47deb P e0f3d400343544cc93a22d3c756e7838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17f0c1fd75e8472680dc5e3148a9c42b" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 33161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:36:38.512274 689 consensus_queue.cc:1035] T 6d4a8e680e9a494ebb52332d9cd47deb P e0f3d400343544cc93a22d3c756e7838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7d3e2f7b2d8486fbd3c67a10f6f9447" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 34633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:36:38.515026 978 mvcc.cc:205] Tried to move back new op lower bound from 7334205024301752320 to 7334205024027107328. Current Snapshot: MvccSnapshot[applied={T|T < 7334205024301752320}]
I20260928 06:36:38.515511 977 mvcc.cc:205] Tried to move back new op lower bound from 7334205024301752320 to 7334205024027107328. Current Snapshot: MvccSnapshot[applied={T|T < 7334205024301752320}]
I20260928 06:36:38.516842 495 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:36:38.516943 495 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:36:38.517743 495 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aae1e59bc3dd4b2c9599a522cbf5b565: Sending DeleteTablet for 3 replicas of tablet 6d4a8e680e9a494ebb52332d9cd47deb
I20260928 06:36:38.518100 603 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d4a8e680e9a494ebb52332d9cd47deb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:36:38 UTC) from {username='kudu', principal='kudu/127.16.157.126@KRBTEST.COM'} at 127.0.0.1:45140
I20260928 06:36:38.518115 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d4a8e680e9a494ebb52332d9cd47deb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:36:38 UTC) from {username='kudu', principal='kudu/127.16.157.126@KRBTEST.COM'} at 127.0.0.1:54358
I20260928 06:36:38.518137 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d4a8e680e9a494ebb52332d9cd47deb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:36:38 UTC) from {username='kudu', principal='kudu/127.16.157.126@KRBTEST.COM'} at 127.0.0.1:34162
I20260928 06:36:38.518323 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 534
I20260928 06:36:38.518371 988 tablet_replica.cc:330] T 6d4a8e680e9a494ebb52332d9cd47deb P 17f0c1fd75e8472680dc5e3148a9c42b: stopping tablet replica
I20260928 06:36:38.518476 988 raft_consensus.cc:2229] T 6d4a8e680e9a494ebb52332d9cd47deb P 17f0c1fd75e8472680dc5e3148a9c42b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:36:38.518577 988 raft_consensus.cc:2258] T 6d4a8e680e9a494ebb52332d9cd47deb P 17f0c1fd75e8472680dc5e3148a9c42b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:36:38.519543 986 tablet_replica.cc:330] T 6d4a8e680e9a494ebb52332d9cd47deb P a7d3e2f7b2d8486fbd3c67a10f6f9447: stopping tablet replica
I20260928 06:36:38.519692 986 raft_consensus.cc:2229] T 6d4a8e680e9a494ebb52332d9cd47deb P a7d3e2f7b2d8486fbd3c67a10f6f9447 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:36:38.519832 986 raft_consensus.cc:2258] T 6d4a8e680e9a494ebb52332d9cd47deb P a7d3e2f7b2d8486fbd3c67a10f6f9447 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:36:38.519874 988 ts_tablet_manager.cc:1905] T 6d4a8e680e9a494ebb52332d9cd47deb P 17f0c1fd75e8472680dc5e3148a9c42b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:36:38.520314 986 ts_tablet_manager.cc:1905] T 6d4a8e680e9a494ebb52332d9cd47deb P a7d3e2f7b2d8486fbd3c67a10f6f9447: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:36:38.521751 988 ts_tablet_manager.cc:1918] T 6d4a8e680e9a494ebb52332d9cd47deb P 17f0c1fd75e8472680dc5e3148a9c42b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:36:38.521835 988 log.cc:1196] T 6d4a8e680e9a494ebb52332d9cd47deb P 17f0c1fd75e8472680dc5e3148a9c42b: Deleting WAL directory at /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-2/wal/wals/6d4a8e680e9a494ebb52332d9cd47deb
I20260928 06:36:38.522121 986 ts_tablet_manager.cc:1918] T 6d4a8e680e9a494ebb52332d9cd47deb P a7d3e2f7b2d8486fbd3c67a10f6f9447: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:36:38.522161 988 ts_tablet_manager.cc:1939] T 6d4a8e680e9a494ebb52332d9cd47deb P 17f0c1fd75e8472680dc5e3148a9c42b: Deleting consensus metadata
I20260928 06:36:38.522233 986 log.cc:1196] T 6d4a8e680e9a494ebb52332d9cd47deb P a7d3e2f7b2d8486fbd3c67a10f6f9447: Deleting WAL directory at /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-1/wal/wals/6d4a8e680e9a494ebb52332d9cd47deb
I20260928 06:36:38.522527 986 ts_tablet_manager.cc:1939] T 6d4a8e680e9a494ebb52332d9cd47deb P a7d3e2f7b2d8486fbd3c67a10f6f9447: Deleting consensus metadata
I20260928 06:36:38.522593 483 catalog_manager.cc:4841] TS 17f0c1fd75e8472680dc5e3148a9c42b (127.16.157.67:33161): tablet 6d4a8e680e9a494ebb52332d9cd47deb (table test-table [id=31746fb73da74f7cb35f6fbb218364bc]) successfully deleted
I20260928 06:36:38.522943 482 catalog_manager.cc:4841] TS a7d3e2f7b2d8486fbd3c67a10f6f9447 (127.16.157.66:34633): tablet 6d4a8e680e9a494ebb52332d9cd47deb (table test-table [id=31746fb73da74f7cb35f6fbb218364bc]) successfully deleted
W20260928 06:36:38.524305 482 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.157.65:45055 (error 108)
W20260928 06:36:38.525163 960 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:38.524635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.65:45055 (local address 127.0.0.1:52152)
0928 06:36:38.524668 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:38.524679 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:38.524723 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:38.525133 (+ 410us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.65:45055: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":16}
I20260928 06:36:38.525384 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 690
W20260928 06:36:38.525321 482 catalog_manager.cc:4584] TS e0f3d400343544cc93a22d3c756e7838 (127.16.157.65:45055): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6d4a8e680e9a494ebb52332d9cd47deb: Network error: Client connection negotiation failed: client connection to 127.16.157.65:45055: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 06:36:38.530354 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 825
I20260928 06:36:38.536964 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 463
I20260928 06:36:58.542181 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:33113
--webserver_interface=127.16.157.126
--webserver_port=40755
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=kudu/127.16.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:43161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.157.126:33113
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:36:58.621703 990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:36:58.621891 990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:36:58.621912 990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:36:58.623188 990 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 06:36:58.623229 990 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 06:36:58.623241 990 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 06:36:58.623253 990 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:36:58.623308 990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:36:58.623327 990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:36:58.623348 990 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:36:58.623373 990 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:36:58.624843 990 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:43161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.16.157.126:33113
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=kudu/127.16.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:33113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.157.126
--webserver_port=40755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:36:58.625098 990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:36:58.625327 990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:36:58.630342 996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:58.630405 998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:58.630482 990 server_base.cc:1034] running on GCE node
W20260928 06:36:58.630414 995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:58.630836 990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:36:58.631191 990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:36:58.632367 990 hybrid_clock.cc:648] HybridClock initialized: now 1790577418632349 us; error 44 us; skew 500 ppm
Sep 28 06:36:58 dist-test-slave-rrj5 krb5kdc[32384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577418, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:36:58.635075 990 init.cc:377] Logged in from keytab as kudu/127.16.157.126@KRBTEST.COM (short username kudu)
I20260928 06:36:58.635493 990 webserver.cc:460] Webserver started at http://127.16.157.126:40755/ using document root <none> and password file <none>
I20260928 06:36:58.635722 990 fs_manager.cc:359] Metadata directory not provided
I20260928 06:36:58.635790 990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:36:58.637135 990 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:36:58.637833 1005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:58.638006 990 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:36:58.638108 990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/master-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/master-0/wal
uuid: "aae1e59bc3dd4b2c9599a522cbf5b565"
format_stamp: "Formatted at 2026-09-28 06:36:16 on dist-test-slave-rrj5"
server_key: "135668f5271aa01d9c65290e12cab450"
server_key_iv: "e688892c5d0ed37454c2914e627fcf4c"
server_key_version: "encryptionkey@0"
I20260928 06:36:58.638441 990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:36:58.664661 990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:36:58.667052 990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:36:58.667237 990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:36:58.671247 990 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.126:33113
I20260928 06:36:58.671303 1057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.126:33113 every 8 connection(s)
I20260928 06:36:58.671674 990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
I20260928 06:36:58.673136 1058 sys_catalog.cc:263] Verifying existing consensus state
I20260928 06:36:58.673826 1058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aae1e59bc3dd4b2c9599a522cbf5b565: Bootstrap starting.
I20260928 06:36:58.677517 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 990
I20260928 06:36:58.677811 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.65:45055
--local_ip_for_outbound_sockets=127.16.157.65
--tserver_master_addrs=127.16.157.126:33113
--webserver_port=35837
--webserver_interface=127.16.157.65
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab
--principal=kudu/127.16.157.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:43161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:36:58.683543 1058 log.cc:824] T 00000000000000000000000000000000 P aae1e59bc3dd4b2c9599a522cbf5b565: Log is configured to *not* fsync() on all Append() calls
I20260928 06:36:58.685901 1058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aae1e59bc3dd4b2c9599a522cbf5b565: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:36:58.686185 1058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aae1e59bc3dd4b2c9599a522cbf5b565: Bootstrap complete.
I20260928 06:36:58.688443 1058 raft_consensus.cc:348] T 00000000000000000000000000000000 P aae1e59bc3dd4b2c9599a522cbf5b565 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aae1e59bc3dd4b2c9599a522cbf5b565" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 33113 } }
I20260928 06:36:58.688697 1058 raft_consensus.cc:729] T 00000000000000000000000000000000 P aae1e59bc3dd4b2c9599a522cbf5b565 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae1e59bc3dd4b2c9599a522cbf5b565, State: Initialized, Role: FOLLOWER
I20260928 06:36:58.688907 1058 consensus_queue.cc:260] T 00000000000000000000000000000000 P aae1e59bc3dd4b2c9599a522cbf5b565 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aae1e59bc3dd4b2c9599a522cbf5b565" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 33113 } }
I20260928 06:36:58.689007 1058 raft_consensus.cc:388] T 00000000000000000000000000000000 P aae1e59bc3dd4b2c9599a522cbf5b565 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:36:58.689062 1058 raft_consensus.cc:482] T 00000000000000000000000000000000 P aae1e59bc3dd4b2c9599a522cbf5b565 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:36:58.689109 1058 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aae1e59bc3dd4b2c9599a522cbf5b565 [term 2 FOLLOWER]: Advancing to term 3
I20260928 06:36:58.690158 1058 raft_consensus.cc:504] T 00000000000000000000000000000000 P aae1e59bc3dd4b2c9599a522cbf5b565 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aae1e59bc3dd4b2c9599a522cbf5b565" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 33113 } }
I20260928 06:36:58.690281 1058 leader_election.cc:302] T 00000000000000000000000000000000 P aae1e59bc3dd4b2c9599a522cbf5b565 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aae1e59bc3dd4b2c9599a522cbf5b565; no voters:
I20260928 06:36:58.690481 1058 leader_election.cc:288] T 00000000000000000000000000000000 P aae1e59bc3dd4b2c9599a522cbf5b565 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 06:36:58.690517 1062 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aae1e59bc3dd4b2c9599a522cbf5b565 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 06:36:58.690644 1062 raft_consensus.cc:686] T 00000000000000000000000000000000 P aae1e59bc3dd4b2c9599a522cbf5b565 [term 3 LEADER]: Becoming Leader. State: Replica: aae1e59bc3dd4b2c9599a522cbf5b565, State: Running, Role: LEADER
I20260928 06:36:58.690733 1062 consensus_queue.cc:237] T 00000000000000000000000000000000 P aae1e59bc3dd4b2c9599a522cbf5b565 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aae1e59bc3dd4b2c9599a522cbf5b565" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 33113 } }
I20260928 06:36:58.690774 1058 sys_catalog.cc:564] T 00000000000000000000000000000000 P aae1e59bc3dd4b2c9599a522cbf5b565 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:36:58.690986 1062 sys_catalog.cc:455] T 00000000000000000000000000000000 P aae1e59bc3dd4b2c9599a522cbf5b565 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aae1e59bc3dd4b2c9599a522cbf5b565. Latest consensus state: current_term: 3 leader_uuid: "aae1e59bc3dd4b2c9599a522cbf5b565" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aae1e59bc3dd4b2c9599a522cbf5b565" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 33113 } } }
I20260928 06:36:58.691073 1062 sys_catalog.cc:458] T 00000000000000000000000000000000 P aae1e59bc3dd4b2c9599a522cbf5b565 [sys.catalog]: This master's current role is: LEADER
I20260928 06:36:58.690989 1063 sys_catalog.cc:455] T 00000000000000000000000000000000 P aae1e59bc3dd4b2c9599a522cbf5b565 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "aae1e59bc3dd4b2c9599a522cbf5b565" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aae1e59bc3dd4b2c9599a522cbf5b565" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 33113 } } }
I20260928 06:36:58.691226 1063 sys_catalog.cc:458] T 00000000000000000000000000000000 P aae1e59bc3dd4b2c9599a522cbf5b565 [sys.catalog]: This master's current role is: LEADER
I20260928 06:36:58.691289 1066 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:36:58.692195 1066 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:36:58.692320 1066 catalog_manager.cc:1259] Loaded cluster ID: 40c2479ccc80479fa303215377f5c4eb
I20260928 06:36:58.692354 1066 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:36:58.693465 1066 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:36:58.693676 1066 catalog_manager.cc:5879] T 00000000000000000000000000000000 P aae1e59bc3dd4b2c9599a522cbf5b565: Loaded TSK: 0
I20260928 06:36:58.693730 1066 catalog_manager.cc:5879] T 00000000000000000000000000000000 P aae1e59bc3dd4b2c9599a522cbf5b565: Loaded TSK: 1
F20260928 06:36:58.693953 1066 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f12792ab9ad google::LogMessage::Fail() at ??:0
@ 0x7f12792ae88a google::LogMessage::SendToLog() at ??:0
@ 0x7f12792ab4de google::LogMessage::Flush() at ??:0
@ 0x7f12792aca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f127ba18104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f127ba15d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f12812051cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f128120548e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f1281205594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f12811cbc48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f12811d5ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f127b1790d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f127b16e954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f127bfd26db start_thread at ??:0
@ 0x7f1277da961f clone at ??:0
W20260928 06:36:58.776706 1060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:36:58.776885 1060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:36:58.776907 1060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:36:58.778160 1060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:36:58.778201 1060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:36:58.778228 1060 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.65
I20260928 06:36:58.779682 1060 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:43161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab
--principal=kudu/127.16.157.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.65:45055
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.157.65
--webserver_port=35837
--tserver_master_addrs=127.16.157.126:33113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1060
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.65
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:36:58.779907 1060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:36:58.780160 1060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:36:58.785702 1099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:58.785714 1100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:58.785714 1102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:58.786115 1060 server_base.cc:1034] running on GCE node
I20260928 06:36:58.786319 1060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:36:58.786559 1060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:36:58.787729 1060 hybrid_clock.cc:648] HybridClock initialized: now 1790577418787713 us; error 34 us; skew 500 ppm
Sep 28 06:36:58 dist-test-slave-rrj5 krb5kdc[32384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577418, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:36:58.790377 1060 init.cc:377] Logged in from keytab as kudu/127.16.157.65@KRBTEST.COM (short username kudu)
I20260928 06:36:58.790930 1060 webserver.cc:460] Webserver started at http://127.16.157.65:35837/ using document root <none> and password file <none>
I20260928 06:36:58.791175 1060 fs_manager.cc:359] Metadata directory not provided
I20260928 06:36:58.791246 1060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:36:58.792636 1060 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:36:58.793460 1109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:58.793663 1060 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:36:58.793752 1060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-0/wal
uuid: "e0f3d400343544cc93a22d3c756e7838"
format_stamp: "Formatted at 2026-09-28 06:36:16 on dist-test-slave-rrj5"
server_key: "efcde22511bade5ec702202e732afb3e"
server_key_iv: "fbe53357b50bc140b9b6685798859930"
server_key_version: "encryptionkey@0"
I20260928 06:36:58.794114 1060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:36:58.806344 1060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:36:58.808526 1060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:36:58.808679 1060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:36:58.808905 1060 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:36:58.809314 1116 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 06:36:58.811266 1060 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 06:36:58.811323 1060 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 06:36:58.811364 1060 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 06:36:58.811986 1060 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 06:36:58.812040 1060 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 06:36:58.812126 1116 tablet_bootstrap.cc:492] T 6d4a8e680e9a494ebb52332d9cd47deb P e0f3d400343544cc93a22d3c756e7838: Bootstrap starting.
I20260928 06:36:58.819527 1060 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.65:45055
I20260928 06:36:58.819885 1060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
I20260928 06:36:58.820480 1223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.65:45055 every 8 connection(s)
I20260928 06:36:58.823324 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 1060
I20260928 06:36:58.823616 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.66:34633
--local_ip_for_outbound_sockets=127.16.157.66
--tserver_master_addrs=127.16.157.126:33113
--webserver_port=34285
--webserver_interface=127.16.157.66
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab
--principal=kudu/127.16.157.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:43161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:36:58.824074 1116 log.cc:824] T 6d4a8e680e9a494ebb52332d9cd47deb P e0f3d400343544cc93a22d3c756e7838: Log is configured to *not* fsync() on all Append() calls
I20260928 06:36:58.826987 1116 tablet_bootstrap.cc:492] T 6d4a8e680e9a494ebb52332d9cd47deb P e0f3d400343544cc93a22d3c756e7838: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:36:58.827309 1116 tablet_bootstrap.cc:492] T 6d4a8e680e9a494ebb52332d9cd47deb P e0f3d400343544cc93a22d3c756e7838: Bootstrap complete.
I20260928 06:36:58.828641 1116 ts_tablet_manager.cc:1397] T 6d4a8e680e9a494ebb52332d9cd47deb P e0f3d400343544cc93a22d3c756e7838: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.005s
I20260928 06:36:58.830138 1116 raft_consensus.cc:348] T 6d4a8e680e9a494ebb52332d9cd47deb P e0f3d400343544cc93a22d3c756e7838 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d3e2f7b2d8486fbd3c67a10f6f9447" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 34633 } } peers { permanent_uuid: "17f0c1fd75e8472680dc5e3148a9c42b" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 33161 } } peers { permanent_uuid: "e0f3d400343544cc93a22d3c756e7838" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 45055 } }
I20260928 06:36:58.830480 1116 raft_consensus.cc:729] T 6d4a8e680e9a494ebb52332d9cd47deb P e0f3d400343544cc93a22d3c756e7838 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0f3d400343544cc93a22d3c756e7838, State: Initialized, Role: FOLLOWER
I20260928 06:36:58.830683 1116 consensus_queue.cc:260] T 6d4a8e680e9a494ebb52332d9cd47deb P e0f3d400343544cc93a22d3c756e7838 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d3e2f7b2d8486fbd3c67a10f6f9447" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 34633 } } peers { permanent_uuid: "17f0c1fd75e8472680dc5e3148a9c42b" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 33161 } } peers { permanent_uuid: "e0f3d400343544cc93a22d3c756e7838" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 45055 } }
I20260928 06:36:58.831007 1116 ts_tablet_manager.cc:1428] T 6d4a8e680e9a494ebb52332d9cd47deb P e0f3d400343544cc93a22d3c756e7838: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
W20260928 06:36:58.896986 1226 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:58.820582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:33113 (local address 127.16.157.65:46251)
0928 06:36:58.820767 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:58.820808 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:58.821249 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:58.896776 (+ 75527us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:33113: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":111,"threads_started":1}
W20260928 06:36:58.897254 1224 heartbeater.cc:644] Failed to heartbeat to 127.16.157.126:33113 (0 consecutive failures): Network error: Failed to ping master at 127.16.157.126:33113: Client connection negotiation failed: client connection to 127.16.157.126:33113: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 06:36:58.897644 1226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:58.897514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:33113 (local address 127.16.157.65:56569)
0928 06:36:58.897558 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:58.897618 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260928 06:36:58.898056 1226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:58.897961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:33113 (local address 127.16.157.65:47651)
0928 06:36:58.898007 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:58.898040 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 06:36:58.898188 1224 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 06:36:58.912587 1228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:36:58.912791 1228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:36:58.912814 1228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:36:58.914098 1228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:36:58.914139 1228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:36:58.914165 1228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.66
I20260928 06:36:58.915546 1228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:43161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab
--principal=kudu/127.16.157.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.66:34633
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.157.66
--webserver_port=34285
--tserver_master_addrs=127.16.157.126:33113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.66
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:36:58.915732 1228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:36:58.915899 1228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:36:58.920765 1234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:58.920769 1237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:58.920882 1235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:58.921088 1228 server_base.cc:1034] running on GCE node
I20260928 06:36:58.921293 1228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:36:58.921535 1228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:36:58.922721 1228 hybrid_clock.cc:648] HybridClock initialized: now 1790577418922709 us; error 31 us; skew 500 ppm
Sep 28 06:36:58 dist-test-slave-rrj5 krb5kdc[32384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577418, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:36:58.925182 1228 init.cc:377] Logged in from keytab as kudu/127.16.157.66@KRBTEST.COM (short username kudu)
I20260928 06:36:58.925621 1228 webserver.cc:460] Webserver started at http://127.16.157.66:34285/ using document root <none> and password file <none>
I20260928 06:36:58.925820 1228 fs_manager.cc:359] Metadata directory not provided
I20260928 06:36:58.925863 1228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:36:58.927114 1228 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 06:36:58.927680 1244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:58.927855 1228 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:58.927954 1228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-1/wal
uuid: "a7d3e2f7b2d8486fbd3c67a10f6f9447"
format_stamp: "Formatted at 2026-09-28 06:36:17 on dist-test-slave-rrj5"
server_key: "b32ea6555f8b48a6cdc0d85f0038d664"
server_key_iv: "e4234dfc0b751e95b47b1c06c45088e9"
server_key_version: "encryptionkey@0"
I20260928 06:36:58.928313 1228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:36:58.953043 1228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:36:58.955163 1228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:36:58.955345 1228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:36:58.955607 1228 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:36:58.955929 1228 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:36:58.955986 1228 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:58.956029 1228 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:36:58.956079 1228 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:58.961975 1228 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.66:34633
I20260928 06:36:58.962018 1357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.66:34633 every 8 connection(s)
I20260928 06:36:58.962409 1228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
I20260928 06:36:58.963482 1360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:58.963116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:33113 (local address 127.16.157.66:33725)
0928 06:36:58.963273 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:58.963342 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":72,"threads_started":1}
I20260928 06:36:58.963801 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 1228
W20260928 06:36:58.963955 1358 heartbeater.cc:644] Failed to heartbeat to 127.16.157.126:33113 (0 consecutive failures): Network error: Failed to ping master at 127.16.157.126:33113: Client connection negotiation failed: client connection to 127.16.157.126:33113: connect: Connection refused (error 111)
I20260928 06:36:58.964076 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.67:33161
--local_ip_for_outbound_sockets=127.16.157.67
--tserver_master_addrs=127.16.157.126:33113
--webserver_port=36887
--webserver_interface=127.16.157.67
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab
--principal=kudu/127.16.157.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:43161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:36:58.964339 1360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:58.964192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:33113 (local address 127.16.157.66:35895)
0928 06:36:58.964277 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:58.964310 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260928 06:36:58.964670 1360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:58.964561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:33113 (local address 127.16.157.66:52927)
0928 06:36:58.964611 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:58.964641 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 06:36:58.964829 1358 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 06:36:59.049319 1361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:36:59.049631 1361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:36:59.049714 1361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:36:59.050949 1361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:36:59.050993 1361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:36:59.051021 1361 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.67
I20260928 06:36:59.052436 1361 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:43161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab
--principal=kudu/127.16.157.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.67:33161
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.157.67
--webserver_port=36887
--tserver_master_addrs=127.16.157.126:33113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.67
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:36:59.052636 1361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:36:59.052871 1361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:36:59.058279 1369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:59.058321 1367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:59.058296 1366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:59.058475 1361 server_base.cc:1034] running on GCE node
I20260928 06:36:59.058694 1361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:36:59.058895 1361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:36:59.060081 1361 hybrid_clock.cc:648] HybridClock initialized: now 1790577419060080 us; error 39 us; skew 500 ppm
Sep 28 06:36:59 dist-test-slave-rrj5 krb5kdc[32384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577419, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:36:59.062664 1361 init.cc:377] Logged in from keytab as kudu/127.16.157.67@KRBTEST.COM (short username kudu)
I20260928 06:36:59.063114 1361 webserver.cc:460] Webserver started at http://127.16.157.67:36887/ using document root <none> and password file <none>
I20260928 06:36:59.063341 1361 fs_manager.cc:359] Metadata directory not provided
I20260928 06:36:59.063410 1361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:36:59.064735 1361 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:36:59.065356 1376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:59.065524 1361 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:36:59.065666 1361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-2/wal
uuid: "17f0c1fd75e8472680dc5e3148a9c42b"
format_stamp: "Formatted at 2026-09-28 06:36:17 on dist-test-slave-rrj5"
server_key: "8a201b81bfc149b840eba2129582b3e7"
server_key_iv: "42b053b2b13010f5bfed8243f82a9c2b"
server_key_version: "encryptionkey@0"
I20260928 06:36:59.066053 1361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:36:59.091149 1361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:36:59.093087 1361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:36:59.093258 1361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:36:59.093513 1361 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:36:59.093863 1361 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:36:59.093919 1361 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:59.093992 1361 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:36:59.094024 1361 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:59.100209 1361 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.67:33161
I20260928 06:36:59.100315 1489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.67:33161 every 8 connection(s)
I20260928 06:36:59.100654 1361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
I20260928 06:36:59.101778 1492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:59.101371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:33113 (local address 127.16.157.67:47477)
0928 06:36:59.101543 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:59.101635 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":84,"threads_started":1}
W20260928 06:36:59.102217 1490 heartbeater.cc:644] Failed to heartbeat to 127.16.157.126:33113 (0 consecutive failures): Network error: Failed to ping master at 127.16.157.126:33113: Client connection negotiation failed: client connection to 127.16.157.126:33113: connect: Connection refused (error 111)
I20260928 06:36:59.102500 1492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:59.102405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:33113 (local address 127.16.157.67:45265)
0928 06:36:59.102457 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:59.102483 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 06:36:59.102814 1492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:59.102739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:33113 (local address 127.16.157.67:37863)
0928 06:36:59.102778 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:59.102800 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 06:36:59.102943 1490 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 06:36:59.110086 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 1361
I20260928 06:36:59.898787 1494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:59.898514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:33113 (local address 127.16.157.65:51289)
0928 06:36:59.898666 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:59.898724 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":56,"threads_started":1}
I20260928 06:36:59.965345 1495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:59.965122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:33113 (local address 127.16.157.66:54473)
0928 06:36:59.965253 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:59.965287 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":66,"threads_started":1}
I20260928 06:37:00.103479 1496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:00.103279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:33113 (local address 127.16.157.67:47667)
0928 06:37:00.103398 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:00.103431 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":56,"threads_started":1}
I20260928 06:37:00.169293 1497 raft_consensus.cc:482] T 6d4a8e680e9a494ebb52332d9cd47deb P e0f3d400343544cc93a22d3c756e7838 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:37:00.169458 1497 raft_consensus.cc:504] T 6d4a8e680e9a494ebb52332d9cd47deb P e0f3d400343544cc93a22d3c756e7838 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d3e2f7b2d8486fbd3c67a10f6f9447" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 34633 } } peers { permanent_uuid: "17f0c1fd75e8472680dc5e3148a9c42b" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 33161 } } peers { permanent_uuid: "e0f3d400343544cc93a22d3c756e7838" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 45055 } }
I20260928 06:37:00.169853 1497 leader_election.cc:288] T 6d4a8e680e9a494ebb52332d9cd47deb P e0f3d400343544cc93a22d3c756e7838 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a7d3e2f7b2d8486fbd3c67a10f6f9447 (127.16.157.66:34633), 17f0c1fd75e8472680dc5e3148a9c42b (127.16.157.67:33161)
Sep 28 06:37:00 dist-test-slave-rrj5 krb5kdc[32384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577418, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.65@KRBTEST.COM for kudu/127.16.157.66@KRBTEST.COM
Sep 28 06:37:00 dist-test-slave-rrj5 krb5kdc[32384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577418, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.65@KRBTEST.COM for kudu/127.16.157.67@KRBTEST.COM
I20260928 06:37:00.180397 1498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:00.170059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.67:33161 (local address 127.16.157.65:32811)
0928 06:37:00.170176 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:00.170186 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:00.170240 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:00.171032 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:00.171035 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:00.171415 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:37:00.171599 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:00.171605 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:00.172893 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:00.172894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:00.173721 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:00.173724 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:00.174315 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:00.174316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:00.174440 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:00.175049 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:37:00.175053 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:37:00.176868 (+ 1815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:37:00.178858 (+ 1990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:00.178859 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:00.178860 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:00.179040 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:00.179291 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:00.179292 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:00.179292 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:00.179364 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:00.179796 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:37:00.179802 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:37:00.179955 (+ 153us) client_negotiation.cc:769] Sending connection context
0928 06:37:00.180073 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 06:37:00.180270 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":120,"thread_start_us":46,"threads_started":1}
I20260928 06:37:00.180410 1500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:00.170131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:60469 (local address 127.16.157.66:34633)
0928 06:37:00.170339 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:00.170348 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:00.170377 (+ 29us) server_negotiation.cc:407] Connection header received
0928 06:37:00.170803 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:00.170805 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:00.170852 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:00.170997 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:00.171729 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:00.172749 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:00.173806 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:00.174219 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:00.174347 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:00.176585 (+ 2238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:00.176589 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:00.176589 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:00.176600 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:00.178547 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:00.178971 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:00.178974 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:00.178974 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:00.179057 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:00.179361 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:00.179362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:00.179362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:00.179693 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:00.179809 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:00.180149 (+ 340us) server_negotiation.cc:299] Negotiation successful
0928 06:37:00.180296 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":77,"threads_started":1}
I20260928 06:37:00.180429 1499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:00.170149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:32811 (local address 127.16.157.67:33161)
0928 06:37:00.170333 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:00.170342 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:00.170371 (+ 29us) server_negotiation.cc:407] Connection header received
0928 06:37:00.170803 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:00.170805 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:00.170844 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:00.171008 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:00.171747 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:00.172762 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:00.173810 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:00.174221 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:00.174341 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:00.176986 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:00.176989 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:00.176989 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:00.176998 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:00.178785 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:00.179113 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:00.179116 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:00.179116 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:00.179227 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:00.179460 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:00.179461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:00.179461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:00.179704 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:00.179809 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:00.180149 (+ 340us) server_negotiation.cc:299] Negotiation successful
0928 06:37:00.180296 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":45,"threads_started":1}
I20260928 06:37:00.180442 1494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:00.170032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.66:34633 (local address 127.16.157.65:60469)
0928 06:37:00.170089 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:00.170100 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:00.170154 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:00.171032 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:00.171035 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:00.171424 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:37:00.171599 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:00.171605 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:00.172858 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:00.172859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:00.173722 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:00.173724 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:00.174313 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:00.174314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:00.174442 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:00.174995 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:37:00.175002 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:37:00.176471 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:37:00.178624 (+ 2153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:00.178630 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:00.178632 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:00.178889 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:00.179147 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:00.179148 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:00.179148 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:00.179288 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:00.179796 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:37:00.179806 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:37:00.179955 (+ 149us) client_negotiation.cc:769] Sending connection context
0928 06:37:00.180073 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 06:37:00.180332 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":154}
W20260928 06:37:00.181654 1112 leader_election.cc:341] T 6d4a8e680e9a494ebb52332d9cd47deb P e0f3d400343544cc93a22d3c756e7838 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a7d3e2f7b2d8486fbd3c67a10f6f9447 (127.16.157.66:34633): Not found: Tablet not found: 6d4a8e680e9a494ebb52332d9cd47deb
W20260928 06:37:00.181767 1113 leader_election.cc:341] T 6d4a8e680e9a494ebb52332d9cd47deb P e0f3d400343544cc93a22d3c756e7838 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 17f0c1fd75e8472680dc5e3148a9c42b (127.16.157.67:33161): Not found: Tablet not found: 6d4a8e680e9a494ebb52332d9cd47deb
I20260928 06:37:00.181803 1113 leader_election.cc:302] T 6d4a8e680e9a494ebb52332d9cd47deb P e0f3d400343544cc93a22d3c756e7838 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0f3d400343544cc93a22d3c756e7838; no voters: 17f0c1fd75e8472680dc5e3148a9c42b, a7d3e2f7b2d8486fbd3c67a10f6f9447
I20260928 06:37:00.181910 1497 raft_consensus.cc:2726] T 6d4a8e680e9a494ebb52332d9cd47deb P e0f3d400343544cc93a22d3c756e7838 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:37:00.899564 1501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:00.899332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:33113 (local address 127.16.157.65:41887)
0928 06:37:00.899465 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:00.899516 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":86,"threads_started":1}
I20260928 06:37:00.966035 1502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:00.965818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:33113 (local address 127.16.157.66:40001)
0928 06:37:00.965938 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:00.965983 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":61,"threads_started":1}
I20260928 06:37:01.104136 1503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:01.103940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:33113 (local address 127.16.157.67:48115)
0928 06:37:01.104061 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:01.104096 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":61,"threads_started":1}
I20260928 06:37:01.900211 1504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:01.900006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:33113 (local address 127.16.157.65:33801)
0928 06:37:01.900131 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:01.900163 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":61,"threads_started":1}
I20260928 06:37:01.966696 1505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:01.966476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:33113 (local address 127.16.157.66:36627)
0928 06:37:01.966616 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:01.966654 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":77,"threads_started":1}
I20260928 06:37:02.078070 1506 raft_consensus.cc:482] T 6d4a8e680e9a494ebb52332d9cd47deb P e0f3d400343544cc93a22d3c756e7838 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:37:02.078187 1506 raft_consensus.cc:504] T 6d4a8e680e9a494ebb52332d9cd47deb P e0f3d400343544cc93a22d3c756e7838 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d3e2f7b2d8486fbd3c67a10f6f9447" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 34633 } } peers { permanent_uuid: "17f0c1fd75e8472680dc5e3148a9c42b" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 33161 } } peers { permanent_uuid: "e0f3d400343544cc93a22d3c756e7838" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 45055 } }
I20260928 06:37:02.078420 1506 leader_election.cc:288] T 6d4a8e680e9a494ebb52332d9cd47deb P e0f3d400343544cc93a22d3c756e7838 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a7d3e2f7b2d8486fbd3c67a10f6f9447 (127.16.157.66:34633), 17f0c1fd75e8472680dc5e3148a9c42b (127.16.157.67:33161)
W20260928 06:37:02.078843 1112 leader_election.cc:341] T 6d4a8e680e9a494ebb52332d9cd47deb P e0f3d400343544cc93a22d3c756e7838 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a7d3e2f7b2d8486fbd3c67a10f6f9447 (127.16.157.66:34633): Not found: Tablet not found: 6d4a8e680e9a494ebb52332d9cd47deb
W20260928 06:37:02.078939 1113 leader_election.cc:341] T 6d4a8e680e9a494ebb52332d9cd47deb P e0f3d400343544cc93a22d3c756e7838 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 17f0c1fd75e8472680dc5e3148a9c42b (127.16.157.67:33161): Not found: Tablet not found: 6d4a8e680e9a494ebb52332d9cd47deb
I20260928 06:37:02.079056 1113 leader_election.cc:302] T 6d4a8e680e9a494ebb52332d9cd47deb P e0f3d400343544cc93a22d3c756e7838 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0f3d400343544cc93a22d3c756e7838; no voters: 17f0c1fd75e8472680dc5e3148a9c42b, a7d3e2f7b2d8486fbd3c67a10f6f9447
I20260928 06:37:02.079154 1506 raft_consensus.cc:2726] T 6d4a8e680e9a494ebb52332d9cd47deb P e0f3d400343544cc93a22d3c756e7838 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:37:02.104777 1507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:02.104569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:33113 (local address 127.16.157.67:43173)
0928 06:37:02.104696 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:02.104736 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":66,"threads_started":1}
I20260928 06:37:02.900869 1508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:02.900661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:33113 (local address 127.16.157.65:53777)
0928 06:37:02.900786 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:02.900822 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":63,"threads_started":1}
I20260928 06:37:02.967362 1509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:02.967159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:33113 (local address 127.16.157.66:60769)
0928 06:37:02.967281 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:02.967319 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":59,"threads_started":1}
I20260928 06:37:03.105515 1510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:03.105313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:33113 (local address 127.16.157.67:48507)
0928 06:37:03.105436 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:03.105470 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":60,"threads_started":1}
I20260928 06:37:03.901726 1511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:03.901433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:33113 (local address 127.16.157.65:52863)
0928 06:37:03.901627 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:03.901675 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":64,"threads_started":1}
I20260928 06:37:03.968072 1512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:03.967859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:33113 (local address 127.16.157.66:42667)
0928 06:37:03.967997 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:03.968028 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":76,"threads_started":1}
I20260928 06:37:04.056476 1513 raft_consensus.cc:482] T 6d4a8e680e9a494ebb52332d9cd47deb P e0f3d400343544cc93a22d3c756e7838 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:37:04.056586 1513 raft_consensus.cc:504] T 6d4a8e680e9a494ebb52332d9cd47deb P e0f3d400343544cc93a22d3c756e7838 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d3e2f7b2d8486fbd3c67a10f6f9447" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 34633 } } peers { permanent_uuid: "17f0c1fd75e8472680dc5e3148a9c42b" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 33161 } } peers { permanent_uuid: "e0f3d400343544cc93a22d3c756e7838" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 45055 } }
I20260928 06:37:04.056826 1513 leader_election.cc:288] T 6d4a8e680e9a494ebb52332d9cd47deb P e0f3d400343544cc93a22d3c756e7838 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a7d3e2f7b2d8486fbd3c67a10f6f9447 (127.16.157.66:34633), 17f0c1fd75e8472680dc5e3148a9c42b (127.16.157.67:33161)
W20260928 06:37:04.057266 1112 leader_election.cc:341] T 6d4a8e680e9a494ebb52332d9cd47deb P e0f3d400343544cc93a22d3c756e7838 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a7d3e2f7b2d8486fbd3c67a10f6f9447 (127.16.157.66:34633): Not found: Tablet not found: 6d4a8e680e9a494ebb52332d9cd47deb
W20260928 06:37:04.057335 1113 leader_election.cc:341] T 6d4a8e680e9a494ebb52332d9cd47deb P e0f3d400343544cc93a22d3c756e7838 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 17f0c1fd75e8472680dc5e3148a9c42b (127.16.157.67:33161): Not found: Tablet not found: 6d4a8e680e9a494ebb52332d9cd47deb
I20260928 06:37:04.057453 1113 leader_election.cc:302] T 6d4a8e680e9a494ebb52332d9cd47deb P e0f3d400343544cc93a22d3c756e7838 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0f3d400343544cc93a22d3c756e7838; no voters: 17f0c1fd75e8472680dc5e3148a9c42b, a7d3e2f7b2d8486fbd3c67a10f6f9447
I20260928 06:37:04.057525 1513 raft_consensus.cc:2726] T 6d4a8e680e9a494ebb52332d9cd47deb P e0f3d400343544cc93a22d3c756e7838 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:37:04.106248 1514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:04.106032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:33113 (local address 127.16.157.67:40883)
0928 06:37:04.106170 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:04.106204 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":79,"threads_started":1}
I20260928 06:37:04.902426 1515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:04.902200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:33113 (local address 127.16.157.65:40505)
0928 06:37:04.902334 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:04.902378 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":85,"threads_started":1}
I20260928 06:37:04.968751 1516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:04.968558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:33113 (local address 127.16.157.66:34029)
0928 06:37:04.968672 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:04.968711 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":72,"threads_started":1}
I20260928 06:37:05.106897 1517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:05.106684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:33113 (local address 127.16.157.67:50309)
0928 06:37:05.106823 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:05.106858 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":81,"threads_started":1}
I20260928 06:37:05.758775 1518 raft_consensus.cc:482] T 6d4a8e680e9a494ebb52332d9cd47deb P e0f3d400343544cc93a22d3c756e7838 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:37:05.758906 1518 raft_consensus.cc:504] T 6d4a8e680e9a494ebb52332d9cd47deb P e0f3d400343544cc93a22d3c756e7838 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d3e2f7b2d8486fbd3c67a10f6f9447" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 34633 } } peers { permanent_uuid: "17f0c1fd75e8472680dc5e3148a9c42b" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 33161 } } peers { permanent_uuid: "e0f3d400343544cc93a22d3c756e7838" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 45055 } }
I20260928 06:37:05.759078 1518 leader_election.cc:288] T 6d4a8e680e9a494ebb52332d9cd47deb P e0f3d400343544cc93a22d3c756e7838 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a7d3e2f7b2d8486fbd3c67a10f6f9447 (127.16.157.66:34633), 17f0c1fd75e8472680dc5e3148a9c42b (127.16.157.67:33161)
W20260928 06:37:05.759444 1112 leader_election.cc:341] T 6d4a8e680e9a494ebb52332d9cd47deb P e0f3d400343544cc93a22d3c756e7838 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a7d3e2f7b2d8486fbd3c67a10f6f9447 (127.16.157.66:34633): Not found: Tablet not found: 6d4a8e680e9a494ebb52332d9cd47deb
W20260928 06:37:05.759552 1113 leader_election.cc:341] T 6d4a8e680e9a494ebb52332d9cd47deb P e0f3d400343544cc93a22d3c756e7838 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 17f0c1fd75e8472680dc5e3148a9c42b (127.16.157.67:33161): Not found: Tablet not found: 6d4a8e680e9a494ebb52332d9cd47deb
I20260928 06:37:05.759622 1113 leader_election.cc:302] T 6d4a8e680e9a494ebb52332d9cd47deb P e0f3d400343544cc93a22d3c756e7838 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0f3d400343544cc93a22d3c756e7838; no voters: 17f0c1fd75e8472680dc5e3148a9c42b, a7d3e2f7b2d8486fbd3c67a10f6f9447
I20260928 06:37:05.759718 1518 raft_consensus.cc:2726] T 6d4a8e680e9a494ebb52332d9cd47deb P e0f3d400343544cc93a22d3c756e7838 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:37:05.903146 1519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:05.902928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:33113 (local address 127.16.157.65:38627)
0928 06:37:05.903056 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:05.903098 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":63,"threads_started":1}
I20260928 06:37:05.969385 1520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:05.969179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:33113 (local address 127.16.157.66:55497)
0928 06:37:05.969307 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:05.969342 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":62,"threads_started":1}
I20260928 06:37:06.107635 1521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:06.107373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:33113 (local address 127.16.157.67:49891)
0928 06:37:06.107538 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:06.107575 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":59,"threads_started":1}
I20260928 06:37:06.904011 1522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:06.903790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:33113 (local address 127.16.157.65:39327)
0928 06:37:06.903926 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:06.903965 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":67,"threads_started":1}
I20260928 06:37:06.970001 1523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:06.969784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:33113 (local address 127.16.157.66:42115)
0928 06:37:06.969931 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:06.969964 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":88,"threads_started":1}
I20260928 06:37:07.108349 1524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:07.108114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:33113 (local address 127.16.157.67:48617)
0928 06:37:07.108254 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:07.108291 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":63,"threads_started":1}
I20260928 06:37:07.904696 1525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:07.904489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:33113 (local address 127.16.157.65:36637)
0928 06:37:07.904613 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:07.904649 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":61,"threads_started":1}
I20260928 06:37:07.970631 1526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:07.970437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:33113 (local address 127.16.157.66:56875)
0928 06:37:07.970555 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:07.970591 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":57,"threads_started":1}
I20260928 06:37:08.109025 1527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:08.108818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:33113 (local address 127.16.157.67:37865)
0928 06:37:08.108945 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:08.108981 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":57,"threads_started":1}
I20260928 06:37:08.905385 1528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:08.905161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:33113 (local address 127.16.157.65:36813)
0928 06:37:08.905305 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:08.905337 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":65,"threads_started":1}
I20260928 06:37:08.971421 1529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:08.971186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:33113 (local address 127.16.157.66:58479)
0928 06:37:08.971320 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:08.971377 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":63,"threads_started":1}
I20260928 06:37:09.109802 1530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:09.109506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:33113 (local address 127.16.157.67:48657)
0928 06:37:09.109713 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:09.109753 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":132,"threads_started":1}
I20260928 06:37:09.906180 1531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:09.905957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:33113 (local address 127.16.157.65:55213)
0928 06:37:09.906090 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:09.906133 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":64,"threads_started":1}
I20260928 06:37:09.972374 1532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:09.972128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:33113 (local address 127.16.157.66:48259)
0928 06:37:09.972279 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:09.972327 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":64,"threads_started":1}
I20260928 06:37:10.110621 1533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:10.110344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:33113 (local address 127.16.157.67:37933)
0928 06:37:10.110512 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:10.110556 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":90,"threads_started":1}
I20260928 06:37:10.906939 1534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:10.906699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:33113 (local address 127.16.157.65:50575)
0928 06:37:10.906846 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:10.906890 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":61,"threads_started":1}
I20260928 06:37:10.973107 1535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:10.972862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:33113 (local address 127.16.157.66:43859)
0928 06:37:10.973017 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:10.973061 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":81,"threads_started":1}
I20260928 06:37:11.111361 1536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:11.111118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:33113 (local address 127.16.157.67:58447)
0928 06:37:11.111270 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:11.111311 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":81,"threads_started":1}
I20260928 06:37:11.635211 1537 raft_consensus.cc:482] T 6d4a8e680e9a494ebb52332d9cd47deb P e0f3d400343544cc93a22d3c756e7838 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:37:11.635308 1537 raft_consensus.cc:504] T 6d4a8e680e9a494ebb52332d9cd47deb P e0f3d400343544cc93a22d3c756e7838 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d3e2f7b2d8486fbd3c67a10f6f9447" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 34633 } } peers { permanent_uuid: "17f0c1fd75e8472680dc5e3148a9c42b" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 33161 } } peers { permanent_uuid: "e0f3d400343544cc93a22d3c756e7838" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 45055 } }
I20260928 06:37:11.635480 1537 leader_election.cc:288] T 6d4a8e680e9a494ebb52332d9cd47deb P e0f3d400343544cc93a22d3c756e7838 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a7d3e2f7b2d8486fbd3c67a10f6f9447 (127.16.157.66:34633), 17f0c1fd75e8472680dc5e3148a9c42b (127.16.157.67:33161)
W20260928 06:37:11.635897 1112 leader_election.cc:341] T 6d4a8e680e9a494ebb52332d9cd47deb P e0f3d400343544cc93a22d3c756e7838 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a7d3e2f7b2d8486fbd3c67a10f6f9447 (127.16.157.66:34633): Not found: Tablet not found: 6d4a8e680e9a494ebb52332d9cd47deb
W20260928 06:37:11.636004 1113 leader_election.cc:341] T 6d4a8e680e9a494ebb52332d9cd47deb P e0f3d400343544cc93a22d3c756e7838 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 17f0c1fd75e8472680dc5e3148a9c42b (127.16.157.67:33161): Not found: Tablet not found: 6d4a8e680e9a494ebb52332d9cd47deb
I20260928 06:37:11.636041 1113 leader_election.cc:302] T 6d4a8e680e9a494ebb52332d9cd47deb P e0f3d400343544cc93a22d3c756e7838 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0f3d400343544cc93a22d3c756e7838; no voters: 17f0c1fd75e8472680dc5e3148a9c42b, a7d3e2f7b2d8486fbd3c67a10f6f9447
I20260928 06:37:11.636159 1537 raft_consensus.cc:2726] T 6d4a8e680e9a494ebb52332d9cd47deb P e0f3d400343544cc93a22d3c756e7838 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:37:11.907817 1538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:11.907529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:33113 (local address 127.16.157.65:44125)
0928 06:37:11.907706 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:11.907750 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":74,"threads_started":1}
I20260928 06:37:11.973824 1539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:11.973606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:33113 (local address 127.16.157.66:34277)
0928 06:37:11.973740 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:11.973778 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":68,"threads_started":1}
I20260928 06:37:12.112133 1540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:12.111910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:33113 (local address 127.16.157.67:58351)
0928 06:37:12.112044 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:12.112084 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":68,"threads_started":1}
I20260928 06:37:12.908681 1541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:12.908387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:33113 (local address 127.16.157.65:41453)
0928 06:37:12.908596 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:12.908633 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":63,"threads_started":1}
I20260928 06:37:12.974453 1542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:12.974240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:33113 (local address 127.16.157.66:37765)
0928 06:37:12.974374 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:12.974409 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":77,"threads_started":1}
I20260928 06:37:13.112903 1543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:13.112660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:33113 (local address 127.16.157.67:53641)
0928 06:37:13.112824 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:13.112860 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":82,"threads_started":1}
I20260928 06:37:13.909500 1544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:13.909265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:33113 (local address 127.16.157.65:32843)
0928 06:37:13.909412 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:13.909453 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":64,"threads_started":1}
I20260928 06:37:13.975188 1545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:13.974970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:33113 (local address 127.16.157.66:54203)
0928 06:37:13.975101 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:13.975141 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":58,"threads_started":1}
I20260928 06:37:14.112180 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 1060
I20260928 06:37:14.113992 1546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:14.113427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:33113 (local address 127.16.157.67:60821)
0928 06:37:14.113894 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:14.113934 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":50,"threads_started":1}
I20260928 06:37:14.117817 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 1228
I20260928 06:37:14.122300 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 1361
2026-09-28T06:37:14Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58684 ms)
[----------] 30 tests from SecurityITest (237184 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:37:14 dist-test-slave-rrj5 krb5kdc[1549](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:37:14 dist-test-slave-rrj5 krb5kdc[1549](info): set up 2 sockets
Sep 28 06:37:14 dist-test-slave-rrj5 krb5kdc[1549](info): commencing operation
krb5kdc: starting...
W20260928 06:37:15.162235 17013 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Sep 28 06:37:15 dist-test-slave-rrj5 krb5kdc[1549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:37:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:37:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.157.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab.
Entry for principal kudu/127.16.157.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab.
I20260928 06:37:15.280094 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:35165
--webserver_interface=127.16.157.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=kudu/127.16.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:44341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.157.126:35165
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:37:15.359741 1562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:15.359921 1562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:15.359941 1562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:15.361099 1562 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:37:15.361138 1562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:15.361152 1562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:15.361164 1562 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:37:15.361213 1562 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:37:15.362627 1562 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:44341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.157.126:35165
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=kudu/127.16.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:35165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:37:15.362869 1562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:15.363082 1562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:37:15.368299 1562 server_base.cc:1034] running on GCE node
W20260928 06:37:15.368414 1570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:15.368455 1567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:15.368484 1568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:15.368825 1562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:15.369114 1562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:15.370294 1562 hybrid_clock.cc:648] HybridClock initialized: now 1790577435370278 us; error 27 us; skew 500 ppm
Sep 28 06:37:15 dist-test-slave-rrj5 krb5kdc[1549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577435, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:15.373054 1562 init.cc:377] Logged in from keytab as kudu/127.16.157.126@KRBTEST.COM (short username kudu)
I20260928 06:37:15.373517 1562 webserver.cc:460] Webserver started at http://127.16.157.126:34623/ using document root <none> and password file <none>
I20260928 06:37:15.373800 1562 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:15.373869 1562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:15.374032 1562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:37:15.375057 1562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/minicluster-data/master-0/data/instance:
uuid: "b33560e1fcdf44afb149a0817f4d0cca"
format_stamp: "Formatted at 2026-09-28 06:37:15 on dist-test-slave-rrj5"
server_key: "367e8f64965de3b798e307645ee42a42"
server_key_iv: "8eea9cf097f36866e974faa729357933"
server_key_version: "encryptionkey@0"
I20260928 06:37:15.375407 1562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/minicluster-data/master-0/wal/instance:
uuid: "b33560e1fcdf44afb149a0817f4d0cca"
format_stamp: "Formatted at 2026-09-28 06:37:15 on dist-test-slave-rrj5"
server_key: "367e8f64965de3b798e307645ee42a42"
server_key_iv: "8eea9cf097f36866e974faa729357933"
server_key_version: "encryptionkey@0"
I20260928 06:37:15.376698 1562 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:37:15.377393 1577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:15.377642 1562 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:37:15.377748 1562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/minicluster-data/master-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/minicluster-data/master-0/wal
uuid: "b33560e1fcdf44afb149a0817f4d0cca"
format_stamp: "Formatted at 2026-09-28 06:37:15 on dist-test-slave-rrj5"
server_key: "367e8f64965de3b798e307645ee42a42"
server_key_iv: "8eea9cf097f36866e974faa729357933"
server_key_version: "encryptionkey@0"
I20260928 06:37:15.377835 1562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:15.400291 1562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:15.402184 1562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:15.402344 1562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:15.406514 1562 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.126:35165
I20260928 06:37:15.406549 1629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.126:35165 every 8 connection(s)
I20260928 06:37:15.406955 1562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
I20260928 06:37:15.407786 1630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:37:15.410014 1630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b33560e1fcdf44afb149a0817f4d0cca: Bootstrap starting.
I20260928 06:37:15.410575 1630 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b33560e1fcdf44afb149a0817f4d0cca: Neither blocks nor log segments found. Creating new log.
I20260928 06:37:15.410877 1630 log.cc:824] T 00000000000000000000000000000000 P b33560e1fcdf44afb149a0817f4d0cca: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:15.411536 1630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b33560e1fcdf44afb149a0817f4d0cca: No bootstrap required, opened a new log
I20260928 06:37:15.413090 1630 raft_consensus.cc:348] T 00000000000000000000000000000000 P b33560e1fcdf44afb149a0817f4d0cca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b33560e1fcdf44afb149a0817f4d0cca" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 35165 } }
I20260928 06:37:15.413220 1630 raft_consensus.cc:374] T 00000000000000000000000000000000 P b33560e1fcdf44afb149a0817f4d0cca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:37:15.413244 1630 raft_consensus.cc:729] T 00000000000000000000000000000000 P b33560e1fcdf44afb149a0817f4d0cca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b33560e1fcdf44afb149a0817f4d0cca, State: Initialized, Role: FOLLOWER
I20260928 06:37:15.413412 1630 consensus_queue.cc:260] T 00000000000000000000000000000000 P b33560e1fcdf44afb149a0817f4d0cca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b33560e1fcdf44afb149a0817f4d0cca" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 35165 } }
I20260928 06:37:15.413477 1630 raft_consensus.cc:388] T 00000000000000000000000000000000 P b33560e1fcdf44afb149a0817f4d0cca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:37:15.413498 1630 raft_consensus.cc:482] T 00000000000000000000000000000000 P b33560e1fcdf44afb149a0817f4d0cca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:37:15.413604 1630 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b33560e1fcdf44afb149a0817f4d0cca [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:37:15.414132 1630 raft_consensus.cc:504] T 00000000000000000000000000000000 P b33560e1fcdf44afb149a0817f4d0cca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b33560e1fcdf44afb149a0817f4d0cca" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 35165 } }
I20260928 06:37:15.414249 1630 leader_election.cc:302] T 00000000000000000000000000000000 P b33560e1fcdf44afb149a0817f4d0cca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b33560e1fcdf44afb149a0817f4d0cca; no voters:
I20260928 06:37:15.414444 1630 leader_election.cc:288] T 00000000000000000000000000000000 P b33560e1fcdf44afb149a0817f4d0cca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:37:15.414491 1633 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b33560e1fcdf44afb149a0817f4d0cca [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:37:15.414696 1630 sys_catalog.cc:564] T 00000000000000000000000000000000 P b33560e1fcdf44afb149a0817f4d0cca [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:37:15.414729 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 1562
I20260928 06:37:15.414844 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/minicluster-data/master-0/wal/instance
I20260928 06:37:15.415043 17013 external_mini_cluster.cc:1428] Setting key 1c54a54ebc77c99db2c92d4e74ce0068
I20260928 06:37:15.415331 1633 raft_consensus.cc:686] T 00000000000000000000000000000000 P b33560e1fcdf44afb149a0817f4d0cca [term 1 LEADER]: Becoming Leader. State: Replica: b33560e1fcdf44afb149a0817f4d0cca, State: Running, Role: LEADER
I20260928 06:37:15.415488 1633 consensus_queue.cc:237] T 00000000000000000000000000000000 P b33560e1fcdf44afb149a0817f4d0cca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b33560e1fcdf44afb149a0817f4d0cca" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 35165 } }
I20260928 06:37:15.416409 1638 sys_catalog.cc:455] T 00000000000000000000000000000000 P b33560e1fcdf44afb149a0817f4d0cca [sys.catalog]: SysCatalogTable state changed. Reason: New leader b33560e1fcdf44afb149a0817f4d0cca. Latest consensus state: current_term: 1 leader_uuid: "b33560e1fcdf44afb149a0817f4d0cca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b33560e1fcdf44afb149a0817f4d0cca" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 35165 } } }
I20260928 06:37:15.416495 1638 sys_catalog.cc:458] T 00000000000000000000000000000000 P b33560e1fcdf44afb149a0817f4d0cca [sys.catalog]: This master's current role is: LEADER
I20260928 06:37:15.416611 1634 sys_catalog.cc:455] T 00000000000000000000000000000000 P b33560e1fcdf44afb149a0817f4d0cca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b33560e1fcdf44afb149a0817f4d0cca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b33560e1fcdf44afb149a0817f4d0cca" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 35165 } } }
I20260928 06:37:15.416697 1634 sys_catalog.cc:458] T 00000000000000000000000000000000 P b33560e1fcdf44afb149a0817f4d0cca [sys.catalog]: This master's current role is: LEADER
W20260928 06:37:15.417268 1649 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b33560e1fcdf44afb149a0817f4d0cca: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:37:15.417336 1649 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:37:15.417680 1650 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:37:15.417889 1650 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:37:15.422019 1650 catalog_manager.cc:1347] Generated new cluster ID: ff5009ddb3294d6ca89561700bc50b19
I20260928 06:37:15.422070 1650 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:37:15 dist-test-slave-rrj5 krb5kdc[1549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:37:15.427708 1642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:15.415496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33778 (local address 127.16.157.126:35165)
0928 06:37:15.415911 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:15.415920 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:15.415961 (+ 41us) server_negotiation.cc:407] Connection header received
0928 06:37:15.416074 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:15.416081 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:15.416688 (+ 607us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:15.416858 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:15.417890 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:15.419120 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:15.420208 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:15.420589 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:15.420661 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:15.423414 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:15.423418 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:15.423419 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:15.423429 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:15.426093 (+ 2664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:15.426473 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:15.426476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:15.426477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:15.426597 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:15.426805 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:15.426806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:15.426806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:15.427122 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:15.427281 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:15.427366 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 06:37:15.427503 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":72,"threads_started":1}
I20260928 06:37:15.432256 1650 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:37:15.432945 1650 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:37:15.438499 1650 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b33560e1fcdf44afb149a0817f4d0cca: Generated new TSK 0
I20260928 06:37:15.438673 1650 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.157.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab.
Entry for principal kudu/127.16.157.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab.
I20260928 06:37:15.468079 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.65:0
--local_ip_for_outbound_sockets=127.16.157.65
--webserver_interface=127.16.157.65
--webserver_port=0
--tserver_master_addrs=127.16.157.126:35165
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab
--principal=kudu/127.16.157.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:44341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:37:15.547271 1656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:15.547451 1656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:15.547474 1656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:15.548576 1656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:15.548619 1656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:15.548669 1656 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.65
I20260928 06:37:15.550076 1656 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:44341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab
--principal=kudu/127.16.157.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.157.65
--webserver_port=0
--tserver_master_addrs=127.16.157.126:35165
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.65
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:37:15.550309 1656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:15.550529 1656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:15.555372 1661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:15.555374 1664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:15.555586 1656 server_base.cc:1034] running on GCE node
W20260928 06:37:15.555445 1662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:15.555827 1656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:15.556018 1656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:15.557175 1656 hybrid_clock.cc:648] HybridClock initialized: now 1790577435557161 us; error 35 us; skew 500 ppm
Sep 28 06:37:15 dist-test-slave-rrj5 krb5kdc[1549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577435, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:15.559769 1656 init.cc:377] Logged in from keytab as kudu/127.16.157.65@KRBTEST.COM (short username kudu)
I20260928 06:37:15.560182 1656 webserver.cc:460] Webserver started at http://127.16.157.65:44987/ using document root <none> and password file <none>
I20260928 06:37:15.560396 1656 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:15.560441 1656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:15.560611 1656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:37:15.561666 1656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/minicluster-data/ts-0/data/instance:
uuid: "fff510eb8df44ec9a9611741b2d78876"
format_stamp: "Formatted at 2026-09-28 06:37:15 on dist-test-slave-rrj5"
server_key: "94844db1e52772bb95c415825fc9efcf"
server_key_iv: "93d512fe2d49484a3076c3e17929e2a6"
server_key_version: "encryptionkey@0"
I20260928 06:37:15.562103 1656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/minicluster-data/ts-0/wal/instance:
uuid: "fff510eb8df44ec9a9611741b2d78876"
format_stamp: "Formatted at 2026-09-28 06:37:15 on dist-test-slave-rrj5"
server_key: "94844db1e52772bb95c415825fc9efcf"
server_key_iv: "93d512fe2d49484a3076c3e17929e2a6"
server_key_version: "encryptionkey@0"
I20260928 06:37:15.563391 1656 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 06:37:15.564073 1671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:15.564256 1656 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:37:15.564347 1656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/minicluster-data/ts-0/wal
uuid: "fff510eb8df44ec9a9611741b2d78876"
format_stamp: "Formatted at 2026-09-28 06:37:15 on dist-test-slave-rrj5"
server_key: "94844db1e52772bb95c415825fc9efcf"
server_key_iv: "93d512fe2d49484a3076c3e17929e2a6"
server_key_version: "encryptionkey@0"
I20260928 06:37:15.564411 1656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:15.581889 1656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:15.583668 1656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:15.583778 1656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:15.583966 1656 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:37:15.584278 1656 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:37:15.584335 1656 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:15.584370 1656 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:37:15.584399 1656 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:15.590196 1656 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.65:34299
I20260928 06:37:15.590243 1784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.65:34299 every 8 connection(s)
I20260928 06:37:15.590552 1656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
I20260928 06:37:15.592515 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 1656
I20260928 06:37:15.592651 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/minicluster-data/ts-0/wal/instance
I20260928 06:37:15.592846 17013 external_mini_cluster.cc:1428] Setting key beae679bcf0d5891bfee3fa875e3c5e5
Sep 28 06:37:15 dist-test-slave-rrj5 krb5kdc[1549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577435, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.65@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:37:15.600348 1642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:15.591271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:43117 (local address 127.16.157.126:35165)
0928 06:37:15.591358 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:15.591360 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:15.591546 (+ 186us) server_negotiation.cc:407] Connection header received
0928 06:37:15.592155 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:15.592157 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:15.592175 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:15.592232 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:15.593413 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:15.594159 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:15.595123 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:15.595355 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:15.595435 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:15.597492 (+ 2057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:15.597493 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:15.597494 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:15.597501 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:15.598994 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:15.599404 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:15.599406 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:15.599406 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:15.599467 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:15.599707 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:15.599708 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:15.599708 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:15.599826 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:15.599899 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:15.600167 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 06:37:15.600259 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 06:37:15.600538 1787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:15.591217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:35165 (local address 127.16.157.65:43117)
0928 06:37:15.591413 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:15.591460 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:15.591994 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:15.592299 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:15.592303 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:15.593050 (+ 747us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:37:15.593321 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:15.593326 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:15.594254 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:15.594255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:15.595028 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:15.595030 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:15.595443 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:15.595445 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:15.595570 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:15.596078 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:37:15.596086 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:37:15.597398 (+ 1312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:37:15.599089 (+ 1691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:15.599096 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:15.599099 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:15.599319 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:15.599538 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:15.599539 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:15.599539 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:15.599644 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:15.599901 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:37:15.599908 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:37:15.600041 (+ 133us) client_negotiation.cc:769] Sending connection context
0928 06:37:15.600144 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 06:37:15.600297 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":100,"threads_started":1}
I20260928 06:37:15.601033 1785 heartbeater.cc:347] Connected to a master server at 127.16.157.126:35165
I20260928 06:37:15.601123 1785 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.16.157.66@KRBTEST.COM; defaulting to no policy
I20260928 06:37:15.601315 1785 heartbeater.cc:511] Master 127.16.157.126:35165 requested a full tablet report, sending...
I20260928 06:37:15.601866 1594 ts_manager.cc:194] Registered new tserver with Master: fff510eb8df44ec9a9611741b2d78876 (127.16.157.65:34299)
I20260928 06:37:15.602501 1594 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.157.65@KRBTEST.COM'} at 127.16.157.65:43117
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab.
Entry for principal kudu/127.16.157.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab.
I20260928 06:37:15.618126 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.66:0
--local_ip_for_outbound_sockets=127.16.157.66
--webserver_interface=127.16.157.66
--webserver_port=0
--tserver_master_addrs=127.16.157.126:35165
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab
--principal=kudu/127.16.157.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:44341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:37:15.698292 1790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:15.698489 1790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:15.698514 1790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:15.699576 1790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:15.699610 1790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:15.699636 1790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.66
I20260928 06:37:15.700999 1790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:44341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab
--principal=kudu/127.16.157.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.157.66
--webserver_port=0
--tserver_master_addrs=127.16.157.126:35165
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.66
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:37:15.701170 1790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:15.701330 1790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:15.706225 1796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:15.706271 1790 server_base.cc:1034] running on GCE node
W20260928 06:37:15.706441 1795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:15.706245 1798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:15.706790 1790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:15.707103 1790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:15.708282 1790 hybrid_clock.cc:648] HybridClock initialized: now 1790577435708265 us; error 32 us; skew 500 ppm
Sep 28 06:37:15 dist-test-slave-rrj5 krb5kdc[1549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577435, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:15.710839 1790 init.cc:377] Logged in from keytab as kudu/127.16.157.66@KRBTEST.COM (short username kudu)
I20260928 06:37:15.711246 1790 webserver.cc:460] Webserver started at http://127.16.157.66:43187/ using document root <none> and password file <none>
I20260928 06:37:15.711472 1790 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:15.711541 1790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:15.711691 1790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:37:15.712689 1790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/minicluster-data/ts-1/data/instance:
uuid: "c721bad7d93d4ed1846d8bf75199e245"
format_stamp: "Formatted at 2026-09-28 06:37:15 on dist-test-slave-rrj5"
server_key: "0570ce68a1f78fcb0e9a1227b84dd2a8"
server_key_iv: "db0a07a0a4f700a5e26bdbddd5f6e2f2"
server_key_version: "encryptionkey@0"
I20260928 06:37:15.713047 1790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/minicluster-data/ts-1/wal/instance:
uuid: "c721bad7d93d4ed1846d8bf75199e245"
format_stamp: "Formatted at 2026-09-28 06:37:15 on dist-test-slave-rrj5"
server_key: "0570ce68a1f78fcb0e9a1227b84dd2a8"
server_key_iv: "db0a07a0a4f700a5e26bdbddd5f6e2f2"
server_key_version: "encryptionkey@0"
I20260928 06:37:15.714483 1790 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 06:37:15.715297 1805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:15.715487 1790 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:37:15.715605 1790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/minicluster-data/ts-1/wal
uuid: "c721bad7d93d4ed1846d8bf75199e245"
format_stamp: "Formatted at 2026-09-28 06:37:15 on dist-test-slave-rrj5"
server_key: "0570ce68a1f78fcb0e9a1227b84dd2a8"
server_key_iv: "db0a07a0a4f700a5e26bdbddd5f6e2f2"
server_key_version: "encryptionkey@0"
I20260928 06:37:15.715695 1790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:15.728994 1790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:15.731264 1790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:15.731433 1790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:15.731653 1790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:37:15.731937 1790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:37:15.731992 1790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:15.732029 1790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:37:15.732059 1790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:15.737913 1790 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.66:33185
I20260928 06:37:15.737970 1918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.66:33185 every 8 connection(s)
I20260928 06:37:15.738267 1790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
I20260928 06:37:15.742349 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 1790
I20260928 06:37:15.742504 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/minicluster-data/ts-1/wal/instance
I20260928 06:37:15.742726 17013 external_mini_cluster.cc:1428] Setting key 2f5ae4428bdda5e124b0380d9267f882
Sep 28 06:37:15 dist-test-slave-rrj5 krb5kdc[1549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577435, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.66@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:37:15.747802 1642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:15.738918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.66:38277 (local address 127.16.157.126:35165)
0928 06:37:15.739005 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:15.739008 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:15.739177 (+ 169us) server_negotiation.cc:407] Connection header received
0928 06:37:15.739688 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:15.739689 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:15.739701 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:15.739768 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:15.740676 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:15.741174 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:15.742233 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:15.742440 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:15.742525 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:15.744799 (+ 2274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:15.744801 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:15.744801 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:15.744809 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:15.746439 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:15.746817 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:15.746818 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:15.746819 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:15.746885 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:15.747126 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:15.747127 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:15.747127 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:15.747277 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:15.747419 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:15.747585 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 06:37:15.747676 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 06:37:15.747830 1921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:15.738865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:35165 (local address 127.16.157.66:38277)
0928 06:37:15.739049 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:15.739094 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:15.739578 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:15.739836 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:15.739840 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:15.740379 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:37:15.740557 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:15.740561 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:15.741276 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:15.741277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:15.742106 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:15.742108 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:15.742551 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:15.742552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:15.742714 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:15.743284 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:37:15.743292 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:37:15.744702 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:37:15.746522 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:15.746528 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:15.746530 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:15.746738 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:15.746957 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:15.746957 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:15.746958 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:15.747057 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:15.747347 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:37:15.747352 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:37:15.747458 (+ 106us) client_negotiation.cc:769] Sending connection context
0928 06:37:15.747536 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 06:37:15.747648 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":95,"threads_started":1}
I20260928 06:37:15.748409 1919 heartbeater.cc:347] Connected to a master server at 127.16.157.126:35165
I20260928 06:37:15.748498 1919 heartbeater.cc:464] Registering TS with master...
I20260928 06:37:15.748678 1919 heartbeater.cc:511] Master 127.16.157.126:35165 requested a full tablet report, sending...
I20260928 06:37:15.749109 1594 ts_manager.cc:194] Registered new tserver with Master: c721bad7d93d4ed1846d8bf75199e245 (127.16.157.66:33185)
I20260928 06:37:15.749498 1594 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.157.66@KRBTEST.COM'} at 127.16.157.66:38277
WARNING: no policy specified for kudu/127.16.157.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab.
Entry for principal kudu/127.16.157.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab.
I20260928 06:37:15.768862 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.67:0
--local_ip_for_outbound_sockets=127.16.157.67
--webserver_interface=127.16.157.67
--webserver_port=0
--tserver_master_addrs=127.16.157.126:35165
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab
--principal=kudu/127.16.157.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:44341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:37:15.850824 1924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:15.851063 1924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:15.851105 1924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:15.852294 1924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:15.852382 1924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:15.852428 1924 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.67
I20260928 06:37:15.853889 1924 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:44341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab
--principal=kudu/127.16.157.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.157.67
--webserver_port=0
--tserver_master_addrs=127.16.157.126:35165
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1924
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.67
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:37:15.854153 1924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:15.854375 1924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:15.859365 1929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:15.859494 1930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:15.859478 1924 server_base.cc:1034] running on GCE node
W20260928 06:37:15.859678 1932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:15.859951 1924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:15.860194 1924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:15.861374 1924 hybrid_clock.cc:648] HybridClock initialized: now 1790577435861370 us; error 31 us; skew 500 ppm
Sep 28 06:37:15 dist-test-slave-rrj5 krb5kdc[1549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577435, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:15.863811 1924 init.cc:377] Logged in from keytab as kudu/127.16.157.67@KRBTEST.COM (short username kudu)
I20260928 06:37:15.864293 1924 webserver.cc:460] Webserver started at http://127.16.157.67:42095/ using document root <none> and password file <none>
I20260928 06:37:15.864540 1924 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:15.864607 1924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:15.864742 1924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:37:15.865806 1924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/minicluster-data/ts-2/data/instance:
uuid: "a9dc2ef5f5b240c0b078ca1917f8369e"
format_stamp: "Formatted at 2026-09-28 06:37:15 on dist-test-slave-rrj5"
server_key: "8e41a507f2475974f71772f1173e0e5d"
server_key_iv: "cb3ef20fe002289731a3b44b4b7aa9da"
server_key_version: "encryptionkey@0"
I20260928 06:37:15.866161 1924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/minicluster-data/ts-2/wal/instance:
uuid: "a9dc2ef5f5b240c0b078ca1917f8369e"
format_stamp: "Formatted at 2026-09-28 06:37:15 on dist-test-slave-rrj5"
server_key: "8e41a507f2475974f71772f1173e0e5d"
server_key_iv: "cb3ef20fe002289731a3b44b4b7aa9da"
server_key_version: "encryptionkey@0"
I20260928 06:37:15.867555 1924 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:37:15.868351 1939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:15.868530 1924 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:37:15.868637 1924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/minicluster-data/ts-2/wal
uuid: "a9dc2ef5f5b240c0b078ca1917f8369e"
format_stamp: "Formatted at 2026-09-28 06:37:15 on dist-test-slave-rrj5"
server_key: "8e41a507f2475974f71772f1173e0e5d"
server_key_iv: "cb3ef20fe002289731a3b44b4b7aa9da"
server_key_version: "encryptionkey@0"
I20260928 06:37:15.868736 1924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:15.884044 1924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:15.885833 1924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:15.886011 1924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:15.886236 1924 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:37:15.886539 1924 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:37:15.886590 1924 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:15.886642 1924 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:37:15.886674 1924 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:15.892861 1924 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.67:44331
I20260928 06:37:15.892910 2052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.67:44331 every 8 connection(s)
I20260928 06:37:15.893227 1924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
I20260928 06:37:15.893430 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 1924
I20260928 06:37:15.893554 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/minicluster-data/ts-2/wal/instance
I20260928 06:37:15.893826 17013 external_mini_cluster.cc:1428] Setting key a46b8f2dd86d735edd3d58db3d142477
Sep 28 06:37:15 dist-test-slave-rrj5 krb5kdc[1549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577435, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.67@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:37:15.902657 1642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:15.894168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:35003 (local address 127.16.157.126:35165)
0928 06:37:15.894237 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:15.894239 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:15.894479 (+ 240us) server_negotiation.cc:407] Connection header received
0928 06:37:15.895027 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:15.895028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:15.895042 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:15.895093 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:15.896072 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:15.896568 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:15.897681 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:15.897889 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:15.897977 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:15.900029 (+ 2052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:15.900030 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:15.900030 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:15.900038 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:15.901233 (+ 1195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:15.901673 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:15.901674 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:15.901675 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:15.901738 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:15.901998 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:15.901999 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:15.901999 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:15.902119 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:15.902196 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:15.902483 (+ 287us) server_negotiation.cc:299] Negotiation successful
0928 06:37:15.902566 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 06:37:15.902865 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:15.894143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:35165 (local address 127.16.157.67:35003)
0928 06:37:15.894332 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:15.894375 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:15.894907 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:15.895163 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:15.895167 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:15.895755 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:37:15.895951 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:15.895956 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:15.896669 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:15.896671 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:15.897528 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:15.897531 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:15.897992 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:15.897994 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:15.898121 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:15.898564 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:37:15.898572 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:37:15.899933 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:37:15.901326 (+ 1393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:15.901335 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:15.901337 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:15.901557 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:15.901807 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:15.901809 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:15.901809 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:15.901921 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:15.902200 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:37:15.902209 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:37:15.902348 (+ 139us) client_negotiation.cc:769] Sending connection context
0928 06:37:15.902460 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 06:37:15.902619 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":89,"threads_started":1}
I20260928 06:37:15.903388 2053 heartbeater.cc:347] Connected to a master server at 127.16.157.126:35165
I20260928 06:37:15.903502 2053 heartbeater.cc:464] Registering TS with master...
I20260928 06:37:15.903676 2053 heartbeater.cc:511] Master 127.16.157.126:35165 requested a full tablet report, sending...
I20260928 06:37:15.904064 1594 ts_manager.cc:194] Registered new tserver with Master: a9dc2ef5f5b240c0b078ca1917f8369e (127.16.157.67:44331)
I20260928 06:37:15.904435 1594 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.157.67@KRBTEST.COM'} at 127.16.157.67:35003
I20260928 06:37:15.905442 17013 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:37:15.912871 1642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:15.906789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33782 (local address 127.16.157.126:35165)
0928 06:37:15.906860 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:15.906861 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:15.906972 (+ 111us) server_negotiation.cc:407] Connection header received
0928 06:37:15.907008 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:15.907008 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:15.907017 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:15.907070 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:15.907569 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:15.908014 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:15.908739 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:15.908972 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:15.909039 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:15.910518 (+ 1479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:15.910519 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:15.910519 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:15.910525 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:15.911859 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:15.912201 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:15.912202 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:15.912202 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:15.912251 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:15.912444 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:15.912445 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:15.912445 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:15.912573 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:15.912645 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:15.912749 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 06:37:15.912813 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 06:37:15.914618 1594 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33782:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:37:15.916337 1594 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:37:15.922497 2062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:15.918568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.65:34299 (local address 127.0.0.1:42694)
0928 06:37:15.918793 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:15.918817 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:15.918859 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:15.919101 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:15.919105 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:15.919107 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:37:15.919195 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:15.919198 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:15.920177 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:15.920178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:15.921154 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:15.921157 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:15.922186 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:15.922187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:15.922268 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:15.922277 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 06:37:15.922371 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 06:37:15.922427 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":48,"threads_started":1}
I20260928 06:37:15.922510 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:15.918706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42694 (local address 127.16.157.65:34299)
0928 06:37:15.918902 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:15.918907 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:15.918918 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:37:15.918963 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:15.918965 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:15.918982 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:15.919084 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:37:15.919293 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:15.920060 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:15.921267 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:15.922030 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:15.922093 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:15.922149 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:15.922389 (+ 240us) server_negotiation.cc:299] Negotiation successful
0928 06:37:15.922428 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":77,"threads_started":1}
I20260928 06:37:15.923012 2066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:15.918886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49546 (local address 127.16.157.67:44331)
0928 06:37:15.919097 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:15.919101 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:15.919129 (+ 28us) server_negotiation.cc:407] Connection header received
0928 06:37:15.919241 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:15.919244 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:15.919259 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:15.919532 (+ 273us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:37:15.919602 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:15.920684 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:15.921935 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:15.922631 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:15.922706 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:15.922756 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:15.922879 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 06:37:15.922919 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":99,"threads_started":1}
I20260928 06:37:15.923190 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:15.918510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.66:33185 (local address 127.0.0.1:43000)
0928 06:37:15.918689 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:15.918731 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:15.918796 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:15.919432 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:15.919433 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:15.919435 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:37:15.919473 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:15.919474 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:15.920818 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:15.920819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:15.921997 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:15.922001 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:15.922899 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:15.922900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:15.922958 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:15.922962 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:37:15.922993 (+ 31us) client_negotiation.cc:240] Negotiation successful
0928 06:37:15.923038 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":44,"thread_start_us":72,"threads_started":1}
I20260928 06:37:15.923303 2063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:15.918625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43000 (local address 127.16.157.66:33185)
0928 06:37:15.919239 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:15.919243 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:15.919253 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:37:15.919319 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:15.919322 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:15.919337 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:15.919542 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:37:15.919614 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:15.920705 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:15.922090 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:15.922799 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:15.923084 (+ 285us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:15.923157 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:15.923184 (+ 27us) server_negotiation.cc:299] Negotiation successful
0928 06:37:15.923226 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":80,"threads_started":1}
I20260928 06:37:15.923609 2064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:15.918729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.67:44331 (local address 127.0.0.1:49546)
0928 06:37:15.919074 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:15.919084 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:15.919143 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:15.919407 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:15.919408 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:15.919410 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:37:15.919456 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:15.919459 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:15.920816 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:15.920817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:15.921855 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:15.921857 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:15.922706 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:15.922707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:15.922776 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:15.922780 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:37:15.922852 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 06:37:15.922897 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":50,"thread_start_us":40,"threads_started":1}
I20260928 06:37:15.923616 1719 tablet_service.cc:1467] Processing CreateTablet for tablet 2fb0b7cb62654a21bc04eb550ecbd143 (DEFAULT_TABLE table=test-table [id=3935e832608e4b1f8ffa2357ebe95430]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:37:15.923981 1719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fb0b7cb62654a21bc04eb550ecbd143. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:37:15.923976 1853 tablet_service.cc:1467] Processing CreateTablet for tablet 2fb0b7cb62654a21bc04eb550ecbd143 (DEFAULT_TABLE table=test-table [id=3935e832608e4b1f8ffa2357ebe95430]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:37:15.924295 1853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fb0b7cb62654a21bc04eb550ecbd143. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:37:15.926178 1987 tablet_service.cc:1467] Processing CreateTablet for tablet 2fb0b7cb62654a21bc04eb550ecbd143 (DEFAULT_TABLE table=test-table [id=3935e832608e4b1f8ffa2357ebe95430]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:37:15.926488 1987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fb0b7cb62654a21bc04eb550ecbd143. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:37:15.927515 2068 tablet_bootstrap.cc:492] T 2fb0b7cb62654a21bc04eb550ecbd143 P c721bad7d93d4ed1846d8bf75199e245: Bootstrap starting.
I20260928 06:37:15.927745 2067 tablet_bootstrap.cc:492] T 2fb0b7cb62654a21bc04eb550ecbd143 P fff510eb8df44ec9a9611741b2d78876: Bootstrap starting.
I20260928 06:37:15.928469 2067 tablet_bootstrap.cc:654] T 2fb0b7cb62654a21bc04eb550ecbd143 P fff510eb8df44ec9a9611741b2d78876: Neither blocks nor log segments found. Creating new log.
I20260928 06:37:15.929239 2067 log.cc:824] T 2fb0b7cb62654a21bc04eb550ecbd143 P fff510eb8df44ec9a9611741b2d78876: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:15.929702 2070 tablet_bootstrap.cc:492] T 2fb0b7cb62654a21bc04eb550ecbd143 P a9dc2ef5f5b240c0b078ca1917f8369e: Bootstrap starting.
I20260928 06:37:15.930092 2067 tablet_bootstrap.cc:492] T 2fb0b7cb62654a21bc04eb550ecbd143 P fff510eb8df44ec9a9611741b2d78876: No bootstrap required, opened a new log
I20260928 06:37:15.930114 2068 tablet_bootstrap.cc:654] T 2fb0b7cb62654a21bc04eb550ecbd143 P c721bad7d93d4ed1846d8bf75199e245: Neither blocks nor log segments found. Creating new log.
I20260928 06:37:15.930169 2067 ts_tablet_manager.cc:1397] T 2fb0b7cb62654a21bc04eb550ecbd143 P fff510eb8df44ec9a9611741b2d78876: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:37:15.930413 2070 tablet_bootstrap.cc:654] T 2fb0b7cb62654a21bc04eb550ecbd143 P a9dc2ef5f5b240c0b078ca1917f8369e: Neither blocks nor log segments found. Creating new log.
I20260928 06:37:15.930706 2068 log.cc:824] T 2fb0b7cb62654a21bc04eb550ecbd143 P c721bad7d93d4ed1846d8bf75199e245: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:15.931015 2070 log.cc:824] T 2fb0b7cb62654a21bc04eb550ecbd143 P a9dc2ef5f5b240c0b078ca1917f8369e: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:15.931444 2068 tablet_bootstrap.cc:492] T 2fb0b7cb62654a21bc04eb550ecbd143 P c721bad7d93d4ed1846d8bf75199e245: No bootstrap required, opened a new log
I20260928 06:37:15.931511 2068 ts_tablet_manager.cc:1397] T 2fb0b7cb62654a21bc04eb550ecbd143 P c721bad7d93d4ed1846d8bf75199e245: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 06:37:15.931767 2070 tablet_bootstrap.cc:492] T 2fb0b7cb62654a21bc04eb550ecbd143 P a9dc2ef5f5b240c0b078ca1917f8369e: No bootstrap required, opened a new log
I20260928 06:37:15.931836 2070 ts_tablet_manager.cc:1397] T 2fb0b7cb62654a21bc04eb550ecbd143 P a9dc2ef5f5b240c0b078ca1917f8369e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 06:37:15.932605 2067 raft_consensus.cc:348] T 2fb0b7cb62654a21bc04eb550ecbd143 P fff510eb8df44ec9a9611741b2d78876 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c721bad7d93d4ed1846d8bf75199e245" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 33185 } } peers { permanent_uuid: "a9dc2ef5f5b240c0b078ca1917f8369e" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 44331 } } peers { permanent_uuid: "fff510eb8df44ec9a9611741b2d78876" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 34299 } }
I20260928 06:37:15.932801 2067 raft_consensus.cc:374] T 2fb0b7cb62654a21bc04eb550ecbd143 P fff510eb8df44ec9a9611741b2d78876 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:37:15.932842 2067 raft_consensus.cc:729] T 2fb0b7cb62654a21bc04eb550ecbd143 P fff510eb8df44ec9a9611741b2d78876 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fff510eb8df44ec9a9611741b2d78876, State: Initialized, Role: FOLLOWER
I20260928 06:37:15.932960 2067 consensus_queue.cc:260] T 2fb0b7cb62654a21bc04eb550ecbd143 P fff510eb8df44ec9a9611741b2d78876 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c721bad7d93d4ed1846d8bf75199e245" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 33185 } } peers { permanent_uuid: "a9dc2ef5f5b240c0b078ca1917f8369e" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 44331 } } peers { permanent_uuid: "fff510eb8df44ec9a9611741b2d78876" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 34299 } }
I20260928 06:37:15.933235 2067 ts_tablet_manager.cc:1428] T 2fb0b7cb62654a21bc04eb550ecbd143 P fff510eb8df44ec9a9611741b2d78876: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:37:15.933362 1785 heartbeater.cc:503] Master 127.16.157.126:35165 was elected leader, sending a full tablet report...
I20260928 06:37:15.933248 2068 raft_consensus.cc:348] T 2fb0b7cb62654a21bc04eb550ecbd143 P c721bad7d93d4ed1846d8bf75199e245 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c721bad7d93d4ed1846d8bf75199e245" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 33185 } } peers { permanent_uuid: "a9dc2ef5f5b240c0b078ca1917f8369e" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 44331 } } peers { permanent_uuid: "fff510eb8df44ec9a9611741b2d78876" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 34299 } }
I20260928 06:37:15.933440 2068 raft_consensus.cc:374] T 2fb0b7cb62654a21bc04eb550ecbd143 P c721bad7d93d4ed1846d8bf75199e245 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:37:15.933486 2068 raft_consensus.cc:729] T 2fb0b7cb62654a21bc04eb550ecbd143 P c721bad7d93d4ed1846d8bf75199e245 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c721bad7d93d4ed1846d8bf75199e245, State: Initialized, Role: FOLLOWER
I20260928 06:37:15.933626 2068 consensus_queue.cc:260] T 2fb0b7cb62654a21bc04eb550ecbd143 P c721bad7d93d4ed1846d8bf75199e245 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c721bad7d93d4ed1846d8bf75199e245" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 33185 } } peers { permanent_uuid: "a9dc2ef5f5b240c0b078ca1917f8369e" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 44331 } } peers { permanent_uuid: "fff510eb8df44ec9a9611741b2d78876" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 34299 } }
I20260928 06:37:15.933903 1919 heartbeater.cc:503] Master 127.16.157.126:35165 was elected leader, sending a full tablet report...
I20260928 06:37:15.934144 2068 ts_tablet_manager.cc:1428] T 2fb0b7cb62654a21bc04eb550ecbd143 P c721bad7d93d4ed1846d8bf75199e245: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260928 06:37:15.934024 2070 raft_consensus.cc:348] T 2fb0b7cb62654a21bc04eb550ecbd143 P a9dc2ef5f5b240c0b078ca1917f8369e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c721bad7d93d4ed1846d8bf75199e245" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 33185 } } peers { permanent_uuid: "a9dc2ef5f5b240c0b078ca1917f8369e" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 44331 } } peers { permanent_uuid: "fff510eb8df44ec9a9611741b2d78876" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 34299 } }
I20260928 06:37:15.934192 2070 raft_consensus.cc:374] T 2fb0b7cb62654a21bc04eb550ecbd143 P a9dc2ef5f5b240c0b078ca1917f8369e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:37:15.934235 2070 raft_consensus.cc:729] T 2fb0b7cb62654a21bc04eb550ecbd143 P a9dc2ef5f5b240c0b078ca1917f8369e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9dc2ef5f5b240c0b078ca1917f8369e, State: Initialized, Role: FOLLOWER
I20260928 06:37:15.934353 2070 consensus_queue.cc:260] T 2fb0b7cb62654a21bc04eb550ecbd143 P a9dc2ef5f5b240c0b078ca1917f8369e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c721bad7d93d4ed1846d8bf75199e245" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 33185 } } peers { permanent_uuid: "a9dc2ef5f5b240c0b078ca1917f8369e" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 44331 } } peers { permanent_uuid: "fff510eb8df44ec9a9611741b2d78876" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 34299 } }
I20260928 06:37:15.934604 2070 ts_tablet_manager.cc:1428] T 2fb0b7cb62654a21bc04eb550ecbd143 P a9dc2ef5f5b240c0b078ca1917f8369e: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260928 06:37:15.934612 2053 heartbeater.cc:503] Master 127.16.157.126:35165 was elected leader, sending a full tablet report...
W20260928 06:37:15.988835 1920 tablet.cc:2369] T 2fb0b7cb62654a21bc04eb550ecbd143 P c721bad7d93d4ed1846d8bf75199e245: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:37:16.091349 1786 tablet.cc:2369] T 2fb0b7cb62654a21bc04eb550ecbd143 P fff510eb8df44ec9a9611741b2d78876: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:37:16.140027 2075 raft_consensus.cc:482] T 2fb0b7cb62654a21bc04eb550ecbd143 P a9dc2ef5f5b240c0b078ca1917f8369e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:37:16.140161 2075 raft_consensus.cc:504] T 2fb0b7cb62654a21bc04eb550ecbd143 P a9dc2ef5f5b240c0b078ca1917f8369e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c721bad7d93d4ed1846d8bf75199e245" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 33185 } } peers { permanent_uuid: "a9dc2ef5f5b240c0b078ca1917f8369e" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 44331 } } peers { permanent_uuid: "fff510eb8df44ec9a9611741b2d78876" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 34299 } }
I20260928 06:37:16.140515 2075 leader_election.cc:288] T 2fb0b7cb62654a21bc04eb550ecbd143 P a9dc2ef5f5b240c0b078ca1917f8369e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c721bad7d93d4ed1846d8bf75199e245 (127.16.157.66:33185), fff510eb8df44ec9a9611741b2d78876 (127.16.157.65:34299)
I20260928 06:37:16.143546 2063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:16.140867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:45281 (local address 127.16.157.66:33185)
0928 06:37:16.140922 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:16.140924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:16.140933 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:37:16.140972 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:16.140972 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:16.140982 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:16.141048 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:37:16.141212 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:16.141763 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:16.142730 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:16.143158 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:16.143238 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:16.143249 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:16.143441 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 06:37:16.143479 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 06:37:16.143548 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:16.140770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.66:33185 (local address 127.16.157.67:45281)
0928 06:37:16.140831 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:16.140843 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:16.140891 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:16.141053 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:16.141054 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:16.141056 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:37:16.141141 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:16.141143 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:16.141844 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:16.141845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:16.142633 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:16.142636 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:16.143256 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:16.143257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:16.143363 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:16.143366 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:37:16.143435 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 06:37:16.143477 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":29}
I20260928 06:37:16.143682 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:16.140845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:40077 (local address 127.16.157.65:34299)
0928 06:37:16.140902 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:16.140905 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:16.141075 (+ 170us) server_negotiation.cc:407] Connection header received
0928 06:37:16.141149 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:16.141151 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:16.141164 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:16.141240 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:37:16.141412 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:16.141907 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:16.142834 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:16.143181 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:16.143263 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:16.143272 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:16.143526 (+ 254us) server_negotiation.cc:299] Negotiation successful
0928 06:37:16.143571 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 06:37:16.143934 2076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:16.140769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.65:34299 (local address 127.16.157.67:40077)
0928 06:37:16.141000 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:16.141009 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:16.141090 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:16.141242 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:16.141243 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:16.141245 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:37:16.141331 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:16.141332 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:16.141994 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:16.141995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:16.142734 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:16.142736 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:16.143289 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:16.143290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:16.143435 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:16.143438 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:37:16.143503 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 06:37:16.143545 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":25,"thread_start_us":120,"threads_started":1}
I20260928 06:37:16.143890 1873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb0b7cb62654a21bc04eb550ecbd143" candidate_uuid: "a9dc2ef5f5b240c0b078ca1917f8369e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c721bad7d93d4ed1846d8bf75199e245" is_pre_election: true
W20260928 06:37:16.144032 2054 tablet.cc:2369] T 2fb0b7cb62654a21bc04eb550ecbd143 P a9dc2ef5f5b240c0b078ca1917f8369e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:37:16.144073 1873 raft_consensus.cc:2454] T 2fb0b7cb62654a21bc04eb550ecbd143 P c721bad7d93d4ed1846d8bf75199e245 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9dc2ef5f5b240c0b078ca1917f8369e in term 0.
I20260928 06:37:16.144280 1739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb0b7cb62654a21bc04eb550ecbd143" candidate_uuid: "a9dc2ef5f5b240c0b078ca1917f8369e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fff510eb8df44ec9a9611741b2d78876" is_pre_election: true
I20260928 06:37:16.144318 1942 leader_election.cc:302] T 2fb0b7cb62654a21bc04eb550ecbd143 P a9dc2ef5f5b240c0b078ca1917f8369e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9dc2ef5f5b240c0b078ca1917f8369e, c721bad7d93d4ed1846d8bf75199e245; no voters:
I20260928 06:37:16.144459 1739 raft_consensus.cc:2454] T 2fb0b7cb62654a21bc04eb550ecbd143 P fff510eb8df44ec9a9611741b2d78876 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9dc2ef5f5b240c0b078ca1917f8369e in term 0.
I20260928 06:37:16.144464 2075 raft_consensus.cc:2781] T 2fb0b7cb62654a21bc04eb550ecbd143 P a9dc2ef5f5b240c0b078ca1917f8369e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:37:16.144532 2075 raft_consensus.cc:482] T 2fb0b7cb62654a21bc04eb550ecbd143 P a9dc2ef5f5b240c0b078ca1917f8369e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:37:16.144572 2075 raft_consensus.cc:3037] T 2fb0b7cb62654a21bc04eb550ecbd143 P a9dc2ef5f5b240c0b078ca1917f8369e [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:37:16.145305 2075 raft_consensus.cc:504] T 2fb0b7cb62654a21bc04eb550ecbd143 P a9dc2ef5f5b240c0b078ca1917f8369e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c721bad7d93d4ed1846d8bf75199e245" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 33185 } } peers { permanent_uuid: "a9dc2ef5f5b240c0b078ca1917f8369e" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 44331 } } peers { permanent_uuid: "fff510eb8df44ec9a9611741b2d78876" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 34299 } }
I20260928 06:37:16.145524 2075 leader_election.cc:288] T 2fb0b7cb62654a21bc04eb550ecbd143 P a9dc2ef5f5b240c0b078ca1917f8369e [CANDIDATE]: Term 1 election: Requested vote from peers c721bad7d93d4ed1846d8bf75199e245 (127.16.157.66:33185), fff510eb8df44ec9a9611741b2d78876 (127.16.157.65:34299)
I20260928 06:37:16.145602 1873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb0b7cb62654a21bc04eb550ecbd143" candidate_uuid: "a9dc2ef5f5b240c0b078ca1917f8369e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c721bad7d93d4ed1846d8bf75199e245"
I20260928 06:37:16.145722 1873 raft_consensus.cc:3037] T 2fb0b7cb62654a21bc04eb550ecbd143 P c721bad7d93d4ed1846d8bf75199e245 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:37:16.145730 1739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb0b7cb62654a21bc04eb550ecbd143" candidate_uuid: "a9dc2ef5f5b240c0b078ca1917f8369e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fff510eb8df44ec9a9611741b2d78876"
I20260928 06:37:16.145816 1739 raft_consensus.cc:3037] T 2fb0b7cb62654a21bc04eb550ecbd143 P fff510eb8df44ec9a9611741b2d78876 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:37:16.146493 1739 raft_consensus.cc:2454] T 2fb0b7cb62654a21bc04eb550ecbd143 P fff510eb8df44ec9a9611741b2d78876 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9dc2ef5f5b240c0b078ca1917f8369e in term 1.
I20260928 06:37:16.146560 1873 raft_consensus.cc:2454] T 2fb0b7cb62654a21bc04eb550ecbd143 P c721bad7d93d4ed1846d8bf75199e245 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9dc2ef5f5b240c0b078ca1917f8369e in term 1.
I20260928 06:37:16.146697 1940 leader_election.cc:302] T 2fb0b7cb62654a21bc04eb550ecbd143 P a9dc2ef5f5b240c0b078ca1917f8369e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9dc2ef5f5b240c0b078ca1917f8369e, fff510eb8df44ec9a9611741b2d78876; no voters:
I20260928 06:37:16.146795 2075 raft_consensus.cc:2781] T 2fb0b7cb62654a21bc04eb550ecbd143 P a9dc2ef5f5b240c0b078ca1917f8369e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:37:16.146966 2075 raft_consensus.cc:686] T 2fb0b7cb62654a21bc04eb550ecbd143 P a9dc2ef5f5b240c0b078ca1917f8369e [term 1 LEADER]: Becoming Leader. State: Replica: a9dc2ef5f5b240c0b078ca1917f8369e, State: Running, Role: LEADER
I20260928 06:37:16.147075 2075 consensus_queue.cc:237] T 2fb0b7cb62654a21bc04eb550ecbd143 P a9dc2ef5f5b240c0b078ca1917f8369e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c721bad7d93d4ed1846d8bf75199e245" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 33185 } } peers { permanent_uuid: "a9dc2ef5f5b240c0b078ca1917f8369e" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 44331 } } peers { permanent_uuid: "fff510eb8df44ec9a9611741b2d78876" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 34299 } }
I20260928 06:37:16.147915 1593 catalog_manager.cc:5495] T 2fb0b7cb62654a21bc04eb550ecbd143 P a9dc2ef5f5b240c0b078ca1917f8369e reported cstate change: term changed from 0 to 1, leader changed from <none> to a9dc2ef5f5b240c0b078ca1917f8369e (127.16.157.67). New cstate: current_term: 1 leader_uuid: "a9dc2ef5f5b240c0b078ca1917f8369e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c721bad7d93d4ed1846d8bf75199e245" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 33185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9dc2ef5f5b240c0b078ca1917f8369e" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 44331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fff510eb8df44ec9a9611741b2d78876" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 34299 } health_report { overall_health: UNKNOWN } } }
I20260928 06:37:16.206229 2066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:16.203941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49560 (local address 127.16.157.67:44331)
0928 06:37:16.203998 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:16.204000 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:16.204009 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:37:16.204059 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:16.204060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:16.204079 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:16.204144 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:37:16.204271 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:16.204744 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:16.205454 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:16.205657 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:16.205709 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:16.205851 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:37:16.205987 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:37:16.206074 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:16.206105 (+ 31us) server_negotiation.cc:299] Negotiation successful
0928 06:37:16.206130 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 06:37:16.208178 1739 raft_consensus.cc:1261] T 2fb0b7cb62654a21bc04eb550ecbd143 P fff510eb8df44ec9a9611741b2d78876 [term 1 FOLLOWER]: Refusing update from remote peer a9dc2ef5f5b240c0b078ca1917f8369e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:37:16.208560 1873 raft_consensus.cc:1261] T 2fb0b7cb62654a21bc04eb550ecbd143 P c721bad7d93d4ed1846d8bf75199e245 [term 1 FOLLOWER]: Refusing update from remote peer a9dc2ef5f5b240c0b078ca1917f8369e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:37:16.208550 2077 consensus_queue.cc:1035] T 2fb0b7cb62654a21bc04eb550ecbd143 P a9dc2ef5f5b240c0b078ca1917f8369e [LEADER]: Connected to new peer: Peer: permanent_uuid: "fff510eb8df44ec9a9611741b2d78876" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 34299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:37:16.208932 2077 consensus_queue.cc:1035] T 2fb0b7cb62654a21bc04eb550ecbd143 P a9dc2ef5f5b240c0b078ca1917f8369e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c721bad7d93d4ed1846d8bf75199e245" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 33185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:37:16.211369 2080 mvcc.cc:205] Tried to move back new op lower bound from 7334205178705551360 to 7334205178459045888. Current Snapshot: MvccSnapshot[applied={T|T < 7334205178705551360}]
I20260928 06:37:16.211463 2083 mvcc.cc:205] Tried to move back new op lower bound from 7334205178705551360 to 7334205178459045888. Current Snapshot: MvccSnapshot[applied={T|T < 7334205178705551360}]
I20260928 06:37:16.211736 2081 mvcc.cc:205] Tried to move back new op lower bound from 7334205178705551360 to 7334205178459045888. Current Snapshot: MvccSnapshot[applied={T|T < 7334205178705551360}]
I20260928 06:37:16.213634 1594 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:37:16.213743 1594 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:37:16.214499 1594 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b33560e1fcdf44afb149a0817f4d0cca: Sending DeleteTablet for 3 replicas of tablet 2fb0b7cb62654a21bc04eb550ecbd143
I20260928 06:37:16.214856 1719 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fb0b7cb62654a21bc04eb550ecbd143 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:37:16 UTC) from {username='kudu', principal='kudu/127.16.157.126@KRBTEST.COM'} at 127.0.0.1:42694
I20260928 06:37:16.214864 1853 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fb0b7cb62654a21bc04eb550ecbd143 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:37:16 UTC) from {username='kudu', principal='kudu/127.16.157.126@KRBTEST.COM'} at 127.0.0.1:43000
I20260928 06:37:16.215130 2090 tablet_replica.cc:330] T 2fb0b7cb62654a21bc04eb550ecbd143 P c721bad7d93d4ed1846d8bf75199e245: stopping tablet replica
I20260928 06:37:16.215143 1987 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fb0b7cb62654a21bc04eb550ecbd143 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:37:16 UTC) from {username='kudu', principal='kudu/127.16.157.126@KRBTEST.COM'} at 127.0.0.1:49546
I20260928 06:37:16.215337 2090 raft_consensus.cc:2229] T 2fb0b7cb62654a21bc04eb550ecbd143 P c721bad7d93d4ed1846d8bf75199e245 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:37:16.215366 2092 tablet_replica.cc:330] T 2fb0b7cb62654a21bc04eb550ecbd143 P a9dc2ef5f5b240c0b078ca1917f8369e: stopping tablet replica
I20260928 06:37:16.215484 2092 raft_consensus.cc:2229] T 2fb0b7cb62654a21bc04eb550ecbd143 P a9dc2ef5f5b240c0b078ca1917f8369e [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:37:16.215528 2091 tablet_replica.cc:330] T 2fb0b7cb62654a21bc04eb550ecbd143 P fff510eb8df44ec9a9611741b2d78876: stopping tablet replica
I20260928 06:37:16.215611 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 1656
I20260928 06:37:16.215600 2092 raft_consensus.cc:2258] T 2fb0b7cb62654a21bc04eb550ecbd143 P a9dc2ef5f5b240c0b078ca1917f8369e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:37:16.215654 2091 raft_consensus.cc:2229] T 2fb0b7cb62654a21bc04eb550ecbd143 P fff510eb8df44ec9a9611741b2d78876 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:37:16.215756 2091 raft_consensus.cc:2258] T 2fb0b7cb62654a21bc04eb550ecbd143 P fff510eb8df44ec9a9611741b2d78876 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:37:16.216642 2092 ts_tablet_manager.cc:1905] T 2fb0b7cb62654a21bc04eb550ecbd143 P a9dc2ef5f5b240c0b078ca1917f8369e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:37:16.218422 2092 ts_tablet_manager.cc:1918] T 2fb0b7cb62654a21bc04eb550ecbd143 P a9dc2ef5f5b240c0b078ca1917f8369e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:37:16.218487 2092 log.cc:1196] T 2fb0b7cb62654a21bc04eb550ecbd143 P a9dc2ef5f5b240c0b078ca1917f8369e: Deleting WAL directory at /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577196946045-17013-0/minicluster-data/ts-2/wal/wals/2fb0b7cb62654a21bc04eb550ecbd143
I20260928 06:37:16.218779 2092 ts_tablet_manager.cc:1939] T 2fb0b7cb62654a21bc04eb550ecbd143 P a9dc2ef5f5b240c0b078ca1917f8369e: Deleting consensus metadata
I20260928 06:37:16.219183 1580 catalog_manager.cc:4841] TS a9dc2ef5f5b240c0b078ca1917f8369e (127.16.157.67:44331): tablet 2fb0b7cb62654a21bc04eb550ecbd143 (table test-table [id=3935e832608e4b1f8ffa2357ebe95430]) successfully deleted
W20260928 06:37:16.222707 1578 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.157.65:34299 (error 108)
W20260928 06:37:16.223233 2064 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:37:16.223093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.65:34299 (local address 127.0.0.1:42696)
0928 06:37:16.223149 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:16.223190 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.65:34299: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 06:37:16.223356 1578 catalog_manager.cc:4584] TS fff510eb8df44ec9a9611741b2d78876 (127.16.157.65:34299): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2fb0b7cb62654a21bc04eb550ecbd143: Network error: Client connection negotiation failed: client connection to 127.16.157.65:34299: connect: Connection reset by peer (error 104)
I20260928 06:37:16.223443 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 1790
I20260928 06:37:16.230554 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 1924
I20260928 06:37:16.230633 2064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:16.230520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.66:33185 (local address 127.0.0.1:43006)
0928 06:37:16.230568 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:16.230603 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.66:33185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260928 06:37:16.235080 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 1562
2026-09-28T06:37:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2113 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T06:37:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:37:16Z Disabled control of system clock
I20260928 06:37:16.262868 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577196946045-17013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577196946045-17013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577196946045-17013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:34053
--webserver_interface=127.16.157.126
--webserver_port=0
--builtin_ntp_servers=127.16.157.84:42027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.157.126:34053
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 06:37:16.342432 2097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:16.342633 2097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:16.342653 2097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:16.343744 2097 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:37:16.343780 2097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:16.343793 2097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:16.343806 2097 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:37:16.343818 2097 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:37:16.345072 2097 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:42027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577196946045-17013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577196946045-17013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.157.126:34053
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:34053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577196946045-17013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:37:16.345253 2097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:16.345487 2097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:16.350215 2102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:16.350273 2103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:16.350258 2105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:16.350569 2097 server_base.cc:1034] running on GCE node
I20260928 06:37:16.350765 2097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:16.351033 2097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:16.352169 2097 hybrid_clock.cc:648] HybridClock initialized: now 1790577436352154 us; error 30 us; skew 500 ppm
I20260928 06:37:16.353317 2097 webserver.cc:460] Webserver started at http://127.16.157.126:39071/ using document root <none> and password file <none>
I20260928 06:37:16.353565 2097 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:16.353686 2097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:16.353845 2097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:37:16.354926 2097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577196946045-17013-0/minicluster-data/master-0/data/instance:
uuid: "a22cbe765eba4705b1b5c271e94b79af"
format_stamp: "Formatted at 2026-09-28 06:37:16 on dist-test-slave-rrj5"
server_key: "958946167558c3441cf5bf37e1813b91"
server_key_iv: "f4b7fdc9d26dee4fcd9b2f024d23bd6f"
server_key_version: "encryptionkey@0"
I20260928 06:37:16.355294 2097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577196946045-17013-0/minicluster-data/master-0/wal/instance:
uuid: "a22cbe765eba4705b1b5c271e94b79af"
format_stamp: "Formatted at 2026-09-28 06:37:16 on dist-test-slave-rrj5"
server_key: "958946167558c3441cf5bf37e1813b91"
server_key_iv: "f4b7fdc9d26dee4fcd9b2f024d23bd6f"
server_key_version: "encryptionkey@0"
I20260928 06:37:16.356624 2097 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:37:16.357318 2111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:16.357488 2097 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:37:16.357604 2097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577196946045-17013-0/minicluster-data/master-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577196946045-17013-0/minicluster-data/master-0/wal
uuid: "a22cbe765eba4705b1b5c271e94b79af"
format_stamp: "Formatted at 2026-09-28 06:37:16 on dist-test-slave-rrj5"
server_key: "958946167558c3441cf5bf37e1813b91"
server_key_iv: "f4b7fdc9d26dee4fcd9b2f024d23bd6f"
server_key_version: "encryptionkey@0"
I20260928 06:37:16.357677 2097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577196946045-17013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577196946045-17013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577196946045-17013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:16.367619 2097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:16.367842 2097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:16.367939 2097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:16.371670 2097 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.126:34053
I20260928 06:37:16.371726 2163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.126:34053 every 8 connection(s)
I20260928 06:37:16.372010 2097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
I20260928 06:37:16.372953 2164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:37:16.375159 2164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a22cbe765eba4705b1b5c271e94b79af: Bootstrap starting.
I20260928 06:37:16.375721 2164 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a22cbe765eba4705b1b5c271e94b79af: Neither blocks nor log segments found. Creating new log.
I20260928 06:37:16.376009 2164 log.cc:824] T 00000000000000000000000000000000 P a22cbe765eba4705b1b5c271e94b79af: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:16.376649 2164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a22cbe765eba4705b1b5c271e94b79af: No bootstrap required, opened a new log
I20260928 06:37:16.376905 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 2097
I20260928 06:37:16.377004 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577196946045-17013-0/minicluster-data/master-0/wal/instance
I20260928 06:37:16.377197 17013 external_mini_cluster.cc:1428] Setting key bfa36c3c5f72e96e36df951dcbab11bb
I20260928 06:37:16.378969 2164 raft_consensus.cc:348] T 00000000000000000000000000000000 P a22cbe765eba4705b1b5c271e94b79af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a22cbe765eba4705b1b5c271e94b79af" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 34053 } }
I20260928 06:37:16.379137 2164 raft_consensus.cc:374] T 00000000000000000000000000000000 P a22cbe765eba4705b1b5c271e94b79af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:37:16.379176 2164 raft_consensus.cc:729] T 00000000000000000000000000000000 P a22cbe765eba4705b1b5c271e94b79af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a22cbe765eba4705b1b5c271e94b79af, State: Initialized, Role: FOLLOWER
I20260928 06:37:16.379315 2164 consensus_queue.cc:260] T 00000000000000000000000000000000 P a22cbe765eba4705b1b5c271e94b79af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a22cbe765eba4705b1b5c271e94b79af" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 34053 } }
I20260928 06:37:16.379420 2164 raft_consensus.cc:388] T 00000000000000000000000000000000 P a22cbe765eba4705b1b5c271e94b79af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:37:16.379454 2164 raft_consensus.cc:482] T 00000000000000000000000000000000 P a22cbe765eba4705b1b5c271e94b79af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:37:16.379499 2164 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a22cbe765eba4705b1b5c271e94b79af [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:37:16.380196 2164 raft_consensus.cc:504] T 00000000000000000000000000000000 P a22cbe765eba4705b1b5c271e94b79af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a22cbe765eba4705b1b5c271e94b79af" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 34053 } }
I20260928 06:37:16.380312 2164 leader_election.cc:302] T 00000000000000000000000000000000 P a22cbe765eba4705b1b5c271e94b79af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a22cbe765eba4705b1b5c271e94b79af; no voters:
I20260928 06:37:16.380466 2164 leader_election.cc:288] T 00000000000000000000000000000000 P a22cbe765eba4705b1b5c271e94b79af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:37:16.380538 2169 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a22cbe765eba4705b1b5c271e94b79af [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:37:16.380685 2169 raft_consensus.cc:686] T 00000000000000000000000000000000 P a22cbe765eba4705b1b5c271e94b79af [term 1 LEADER]: Becoming Leader. State: Replica: a22cbe765eba4705b1b5c271e94b79af, State: Running, Role: LEADER
I20260928 06:37:16.380702 2164 sys_catalog.cc:564] T 00000000000000000000000000000000 P a22cbe765eba4705b1b5c271e94b79af [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:37:16.380776 2169 consensus_queue.cc:237] T 00000000000000000000000000000000 P a22cbe765eba4705b1b5c271e94b79af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a22cbe765eba4705b1b5c271e94b79af" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 34053 } }
I20260928 06:37:16.381072 2169 sys_catalog.cc:455] T 00000000000000000000000000000000 P a22cbe765eba4705b1b5c271e94b79af [sys.catalog]: SysCatalogTable state changed. Reason: New leader a22cbe765eba4705b1b5c271e94b79af. Latest consensus state: current_term: 1 leader_uuid: "a22cbe765eba4705b1b5c271e94b79af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a22cbe765eba4705b1b5c271e94b79af" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 34053 } } }
I20260928 06:37:16.381155 2169 sys_catalog.cc:458] T 00000000000000000000000000000000 P a22cbe765eba4705b1b5c271e94b79af [sys.catalog]: This master's current role is: LEADER
I20260928 06:37:16.381107 2170 sys_catalog.cc:455] T 00000000000000000000000000000000 P a22cbe765eba4705b1b5c271e94b79af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a22cbe765eba4705b1b5c271e94b79af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a22cbe765eba4705b1b5c271e94b79af" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 34053 } } }
I20260928 06:37:16.381316 2170 sys_catalog.cc:458] T 00000000000000000000000000000000 P a22cbe765eba4705b1b5c271e94b79af [sys.catalog]: This master's current role is: LEADER
I20260928 06:37:16.381358 2176 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:37:16.381902 2176 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:37:16.383610 2176 catalog_manager.cc:1347] Generated new cluster ID: ef6d7fe09a1949769c8f96ad9ae24aaa
I20260928 06:37:16.383672 2176 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:37:16.384878 2168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:16.377765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59808 (local address 127.16.157.126:34053)
0928 06:37:16.377981 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:16.377989 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:16.378039 (+ 50us) server_negotiation.cc:407] Connection header received
0928 06:37:16.378937 (+ 898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:16.378945 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:16.379023 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:16.379175 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:16.379453 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:16.381280 (+ 1827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:16.382601 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:16.383177 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:16.383292 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:16.384074 (+ 782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:16.384078 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:16.384078 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:37:16.384278 (+ 200us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:16.384321 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:37:16.384335 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:16.384406 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:16.384474 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 06:37:16.384654 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":80,"threads_started":1}
I20260928 06:37:16.392555 2176 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:37:16.393211 2176 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:37:16.397328 2176 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a22cbe765eba4705b1b5c271e94b79af: Generated new TSK 0
I20260928 06:37:16.397516 2176 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:37:16.403712 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577196946045-17013-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577196946045-17013-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577196946045-17013-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.65:0
--local_ip_for_outbound_sockets=127.16.157.65
--webserver_interface=127.16.157.65
--webserver_port=0
--tserver_master_addrs=127.16.157.126:34053
--builtin_ntp_servers=127.16.157.84:42027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 06:37:16.484854 2188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:16.485076 2188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:16.485113 2188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:16.486320 2188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:16.486380 2188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:16.486434 2188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.65
I20260928 06:37:16.487756 2188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:42027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577196946045-17013-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577196946045-17013-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.157.65
--webserver_port=0
--tserver_master_addrs=127.16.157.126:34053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.65
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577196946045-17013-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:37:16.488005 2188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:16.488327 2188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:16.494316 2196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:16.494325 2194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:16.494498 2193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:16.494535 2188 server_base.cc:1034] running on GCE node
I20260928 06:37:16.494845 2188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:16.495123 2188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:16.496290 2188 hybrid_clock.cc:648] HybridClock initialized: now 1790577436496282 us; error 36 us; skew 500 ppm
I20260928 06:37:16.497783 2188 webserver.cc:460] Webserver started at http://127.16.157.65:35831/ using document root <none> and password file <none>
I20260928 06:37:16.498059 2188 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:16.498124 2188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:16.498250 2188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:37:16.499647 2188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577196946045-17013-0/minicluster-data/ts-0/data/instance:
uuid: "9efb92170945411286d0c19a07906918"
format_stamp: "Formatted at 2026-09-28 06:37:16 on dist-test-slave-rrj5"
server_key: "02e63c45ffcfb976c50371a3d7d258c3"
server_key_iv: "c57af559a296acc28c42a4bd33a69649"
server_key_version: "encryptionkey@0"
I20260928 06:37:16.500039 2188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577196946045-17013-0/minicluster-data/ts-0/wal/instance:
uuid: "9efb92170945411286d0c19a07906918"
format_stamp: "Formatted at 2026-09-28 06:37:16 on dist-test-slave-rrj5"
server_key: "02e63c45ffcfb976c50371a3d7d258c3"
server_key_iv: "c57af559a296acc28c42a4bd33a69649"
server_key_version: "encryptionkey@0"
I20260928 06:37:16.501695 2188 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:37:16.502609 2202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:16.502849 2188 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:37:16.502965 2188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577196946045-17013-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577196946045-17013-0/minicluster-data/ts-0/wal
uuid: "9efb92170945411286d0c19a07906918"
format_stamp: "Formatted at 2026-09-28 06:37:16 on dist-test-slave-rrj5"
server_key: "02e63c45ffcfb976c50371a3d7d258c3"
server_key_iv: "c57af559a296acc28c42a4bd33a69649"
server_key_version: "encryptionkey@0"
I20260928 06:37:16.503064 2188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577196946045-17013-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577196946045-17013-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577196946045-17013-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:16.517854 2188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:16.518069 2188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:16.518172 2188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:16.518340 2188 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:37:16.518574 2188 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:37:16.518602 2188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:16.518623 2188 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:37:16.518646 2188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:16.524283 2188 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.65:35789
I20260928 06:37:16.524389 2315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.65:35789 every 8 connection(s)
I20260928 06:37:16.524672 2188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
I20260928 06:37:16.528733 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 2188
I20260928 06:37:16.528848 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577196946045-17013-0/minicluster-data/ts-0/wal/instance
I20260928 06:37:16.529063 17013 external_mini_cluster.cc:1428] Setting key 28cc166fd5e5935cef295b89fdf872e9
I20260928 06:37:16.529772 2168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:16.525398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:44749 (local address 127.16.157.126:34053)
0928 06:37:16.525471 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:16.525473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:16.525739 (+ 266us) server_negotiation.cc:407] Connection header received
0928 06:37:16.526198 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:16.526199 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:16.526209 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:16.526258 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:16.526896 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:16.527382 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:16.528281 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:16.528456 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:16.528505 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:16.529493 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:16.529494 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:16.529494 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:37:16.529497 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:16.529511 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:37:16.529514 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:16.529551 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:16.529654 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 06:37:16.529687 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 06:37:16.529976 2318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:16.525385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:34053 (local address 127.16.157.65:44749)
0928 06:37:16.525606 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:16.525645 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:16.526101 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:16.526313 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:16.526316 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:16.526323 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:37:16.526820 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:16.526824 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:16.527464 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:16.527464 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:16.528208 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:16.528210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:16.528528 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:16.528529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:16.528721 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:16.529352 (+ 631us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:37:16.529352 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:37:16.529390 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:37:16.529416 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:37:16.529561 (+ 145us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:37:16.529564 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:37:16.529591 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 06:37:16.529654 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 06:37:16.529751 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":100,"threads_started":1}
I20260928 06:37:16.530200 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577196946045-17013-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577196946045-17013-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577196946045-17013-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.66:0
--local_ip_for_outbound_sockets=127.16.157.66
--webserver_interface=127.16.157.66
--webserver_port=0
--tserver_master_addrs=127.16.157.126:34053
--builtin_ntp_servers=127.16.157.84:42027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 06:37:16.531697 2316 heartbeater.cc:347] Connected to a master server at 127.16.157.126:34053
I20260928 06:37:16.531774 2316 heartbeater.cc:464] Registering TS with master...
I20260928 06:37:16.531992 2316 heartbeater.cc:511] Master 127.16.157.126:34053 requested a full tablet report, sending...
I20260928 06:37:16.532541 2128 ts_manager.cc:194] Registered new tserver with Master: 9efb92170945411286d0c19a07906918 (127.16.157.65:35789)
I20260928 06:37:16.533505 2128 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.157.65:44749
W20260928 06:37:16.611609 2319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:16.611797 2319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:16.611819 2319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:16.612963 2319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:16.613005 2319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:16.613029 2319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.66
I20260928 06:37:16.614367 2319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:42027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577196946045-17013-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577196946045-17013-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.157.66
--webserver_port=0
--tserver_master_addrs=127.16.157.126:34053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.66
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577196946045-17013-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:37:16.614614 2319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:16.614967 2319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:16.619958 2324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:16.619973 2327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:16.619988 2325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:16.620018 2319 server_base.cc:1034] running on GCE node
I20260928 06:37:16.620318 2319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:16.620524 2319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:16.621632 2319 hybrid_clock.cc:648] HybridClock initialized: now 1790577436621614 us; error 51 us; skew 500 ppm
I20260928 06:37:16.622870 2319 webserver.cc:460] Webserver started at http://127.16.157.66:46593/ using document root <none> and password file <none>
I20260928 06:37:16.623107 2319 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:16.623154 2319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:16.623323 2319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:37:16.624356 2319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577196946045-17013-0/minicluster-data/ts-1/data/instance:
uuid: "b372b04e434f448cab24f2eaad618642"
format_stamp: "Formatted at 2026-09-28 06:37:16 on dist-test-slave-rrj5"
server_key: "c3bf46c7751fbf9f911a2c970a9e422a"
server_key_iv: "00cb513ecc3233c6d3ba9e003edcb292"
server_key_version: "encryptionkey@0"
I20260928 06:37:16.624692 2319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577196946045-17013-0/minicluster-data/ts-1/wal/instance:
uuid: "b372b04e434f448cab24f2eaad618642"
format_stamp: "Formatted at 2026-09-28 06:37:16 on dist-test-slave-rrj5"
server_key: "c3bf46c7751fbf9f911a2c970a9e422a"
server_key_iv: "00cb513ecc3233c6d3ba9e003edcb292"
server_key_version: "encryptionkey@0"
I20260928 06:37:16.626039 2319 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:37:16.626708 2333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:16.626874 2319 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:37:16.626965 2319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577196946045-17013-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577196946045-17013-0/minicluster-data/ts-1/wal
uuid: "b372b04e434f448cab24f2eaad618642"
format_stamp: "Formatted at 2026-09-28 06:37:16 on dist-test-slave-rrj5"
server_key: "c3bf46c7751fbf9f911a2c970a9e422a"
server_key_iv: "00cb513ecc3233c6d3ba9e003edcb292"
server_key_version: "encryptionkey@0"
I20260928 06:37:16.627038 2319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577196946045-17013-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577196946045-17013-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577196946045-17013-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:16.658608 2319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:16.658926 2319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:16.659071 2319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:16.659308 2319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:37:16.659632 2319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:37:16.659684 2319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:16.659719 2319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:37:16.659744 2319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:16.666394 2319 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.66:32999
I20260928 06:37:16.666483 2446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.66:32999 every 8 connection(s)
I20260928 06:37:16.666822 2319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
I20260928 06:37:16.671706 2168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:16.667504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.66:55415 (local address 127.16.157.126:34053)
0928 06:37:16.667572 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:16.667574 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:16.667797 (+ 223us) server_negotiation.cc:407] Connection header received
0928 06:37:16.668286 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:16.668287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:16.668297 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:16.668347 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:16.669005 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:16.669493 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:16.670479 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:16.670660 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:16.670729 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:16.671487 (+ 758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:16.671488 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:16.671489 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:37:16.671492 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:16.671506 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:37:16.671510 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:16.671550 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:16.671628 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 06:37:16.671651 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 06:37:16.671948 2449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:16.667512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:34053 (local address 127.16.157.66:55415)
0928 06:37:16.667692 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:16.667724 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:16.668200 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:16.668401 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:16.668405 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:16.668413 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:37:16.668938 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:16.668942 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:16.669608 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:16.669609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:16.670409 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:16.670411 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:16.670753 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:16.670753 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:16.670868 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:16.671368 (+ 500us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:37:16.671369 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:37:16.671395 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:37:16.671416 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:37:16.671563 (+ 147us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:37:16.671567 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:37:16.671569 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:37:16.671620 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 06:37:16.671697 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":82,"threads_started":1}
I20260928 06:37:16.672493 2447 heartbeater.cc:347] Connected to a master server at 127.16.157.126:34053
I20260928 06:37:16.672595 2447 heartbeater.cc:464] Registering TS with master...
I20260928 06:37:16.672767 2447 heartbeater.cc:511] Master 127.16.157.126:34053 requested a full tablet report, sending...
I20260928 06:37:16.673144 2128 ts_manager.cc:194] Registered new tserver with Master: b372b04e434f448cab24f2eaad618642 (127.16.157.66:32999)
I20260928 06:37:16.673504 2128 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.157.66:55415
I20260928 06:37:16.675722 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 2319
I20260928 06:37:16.675860 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577196946045-17013-0/minicluster-data/ts-1/wal/instance
I20260928 06:37:16.676069 17013 external_mini_cluster.cc:1428] Setting key e9956ced5f3595b5bb3006bd20b46800
I20260928 06:37:16.676898 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577196946045-17013-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577196946045-17013-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577196946045-17013-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.67:0
--local_ip_for_outbound_sockets=127.16.157.67
--webserver_interface=127.16.157.67
--webserver_port=0
--tserver_master_addrs=127.16.157.126:34053
--builtin_ntp_servers=127.16.157.84:42027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 06:37:16.756249 2450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:16.756429 2450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:16.756450 2450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:16.757531 2450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:16.757593 2450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:16.757649 2450 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.67
I20260928 06:37:16.758988 2450 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:42027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577196946045-17013-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577196946045-17013-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.157.67
--webserver_port=0
--tserver_master_addrs=127.16.157.126:34053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.67
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577196946045-17013-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:37:16.759204 2450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:16.759490 2450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:16.764588 2455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:16.764611 2456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:16.764611 2458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:16.764760 2450 server_base.cc:1034] running on GCE node
I20260928 06:37:16.764945 2450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:16.765123 2450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:16.766273 2450 hybrid_clock.cc:648] HybridClock initialized: now 1790577436766247 us; error 42 us; skew 500 ppm
I20260928 06:37:16.767400 2450 webserver.cc:460] Webserver started at http://127.16.157.67:38659/ using document root <none> and password file <none>
I20260928 06:37:16.767633 2450 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:16.767678 2450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:16.767853 2450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:37:16.768849 2450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577196946045-17013-0/minicluster-data/ts-2/data/instance:
uuid: "419b1305a17b48b5ac381396ee175e9c"
format_stamp: "Formatted at 2026-09-28 06:37:16 on dist-test-slave-rrj5"
server_key: "f02a9810df7250c6a8776b1c993e3cbd"
server_key_iv: "c83999cbfd8f2516aab2e01fb3dbb9ef"
server_key_version: "encryptionkey@0"
I20260928 06:37:16.769215 2450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577196946045-17013-0/minicluster-data/ts-2/wal/instance:
uuid: "419b1305a17b48b5ac381396ee175e9c"
format_stamp: "Formatted at 2026-09-28 06:37:16 on dist-test-slave-rrj5"
server_key: "f02a9810df7250c6a8776b1c993e3cbd"
server_key_iv: "c83999cbfd8f2516aab2e01fb3dbb9ef"
server_key_version: "encryptionkey@0"
I20260928 06:37:16.770539 2450 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:37:16.771253 2464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:16.771428 2450 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:37:16.771526 2450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577196946045-17013-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577196946045-17013-0/minicluster-data/ts-2/wal
uuid: "419b1305a17b48b5ac381396ee175e9c"
format_stamp: "Formatted at 2026-09-28 06:37:16 on dist-test-slave-rrj5"
server_key: "f02a9810df7250c6a8776b1c993e3cbd"
server_key_iv: "c83999cbfd8f2516aab2e01fb3dbb9ef"
server_key_version: "encryptionkey@0"
I20260928 06:37:16.771608 2450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577196946045-17013-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577196946045-17013-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577196946045-17013-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:16.785828 2450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:16.786115 2450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:16.786264 2450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:16.786484 2450 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:37:16.786795 2450 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:37:16.786850 2450 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:16.786887 2450 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:37:16.786916 2450 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:16.793735 2450 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.67:39561
I20260928 06:37:16.793780 2577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.67:39561 every 8 connection(s)
I20260928 06:37:16.794080 2450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
I20260928 06:37:16.799288 2168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:16.794793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:34757 (local address 127.16.157.126:34053)
0928 06:37:16.794859 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:16.794862 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:16.795093 (+ 231us) server_negotiation.cc:407] Connection header received
0928 06:37:16.795614 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:16.795615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:16.795627 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:16.795684 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:16.796338 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:16.796845 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:16.797909 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:16.798116 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:16.798199 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:16.798940 (+ 741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:16.798941 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:16.798941 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:37:16.798946 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:16.798977 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:37:16.798981 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:16.799051 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:16.799153 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 06:37:16.799183 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 06:37:16.799472 2580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:16.794787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:34053 (local address 127.16.157.67:34757)
0928 06:37:16.794970 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:16.795002 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:16.795504 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:16.795763 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:16.795766 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:16.795775 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:37:16.796239 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:16.796243 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:16.796944 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:16.796945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:16.797812 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:16.797814 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:16.798210 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:16.798211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:16.798348 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:16.798795 (+ 447us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:37:16.798796 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:37:16.798821 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:37:16.798842 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:37:16.799051 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:37:16.799058 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:37:16.799061 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:37:16.799142 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 06:37:16.799243 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":74,"threads_started":1}
I20260928 06:37:16.800050 2578 heartbeater.cc:347] Connected to a master server at 127.16.157.126:34053
I20260928 06:37:16.800143 2578 heartbeater.cc:464] Registering TS with master...
I20260928 06:37:16.800323 2578 heartbeater.cc:511] Master 127.16.157.126:34053 requested a full tablet report, sending...
I20260928 06:37:16.800729 2128 ts_manager.cc:194] Registered new tserver with Master: 419b1305a17b48b5ac381396ee175e9c (127.16.157.67:39561)
I20260928 06:37:16.801090 2128 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.157.67:34757
I20260928 06:37:16.801661 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 2450
I20260928 06:37:16.801765 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577196946045-17013-0/minicluster-data/ts-2/wal/instance
I20260928 06:37:16.801970 17013 external_mini_cluster.cc:1428] Setting key da00b23af5587aec825d4136b3141697
I20260928 06:37:16.802793 17013 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:37:16.806885 2168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:16.804050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59818 (local address 127.16.157.126:34053)
0928 06:37:16.804088 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:16.804091 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:16.804200 (+ 109us) server_negotiation.cc:407] Connection header received
0928 06:37:16.804280 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:16.804281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:16.804292 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:16.804336 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:16.804501 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:16.804958 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:16.805781 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:16.805984 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:16.806057 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:16.806633 (+ 576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:16.806634 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:16.806634 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:37:16.806639 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:16.806650 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:37:16.806653 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:16.806731 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:16.806788 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 06:37:16.806808 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 06:37:16.808308 2128 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59818:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:37:16.809861 2128 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:37:16.815574 2587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:16.811874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.67:39561 (local address 127.0.0.1:37560)
0928 06:37:16.812017 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:16.812054 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:16.812118 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:16.812430 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:16.812434 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:16.812437 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:37:16.812501 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:16.812503 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:16.813536 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:16.813537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:16.814529 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:16.814531 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:16.815286 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:16.815287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:16.815360 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:16.815373 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 06:37:16.815442 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 06:37:16.815488 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":56,"threads_started":1}
I20260928 06:37:16.815615 2588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:16.811973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37560 (local address 127.16.157.67:39561)
0928 06:37:16.812235 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:16.812239 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:16.812250 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:37:16.812291 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:16.812293 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:16.812308 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:16.812579 (+ 271us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:37:16.812625 (+ 46us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:16.813446 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:16.814604 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:16.815190 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:16.815265 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:16.815357 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:16.815464 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 06:37:16.815502 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":110,"threads_started":1}
I20260928 06:37:16.816375 2501 tablet_service.cc:1467] Processing CreateTablet for tablet 57ac801409aa4163aed7a410b139310e (DEFAULT_TABLE table=test-table [id=3bae3d683f20448dbb47d42b094fab08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:37:16.816589 2589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:16.812083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.66:32999 (local address 127.0.0.1:39238)
0928 06:37:16.812453 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:16.812464 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:16.812519 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:16.812755 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:16.812756 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:16.812758 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:37:16.812803 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:16.812804 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:16.814186 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:16.814187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:16.815092 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:16.815094 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:16.816369 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:16.816370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:16.816427 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:16.816431 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:37:16.816497 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 06:37:16.816528 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":79,"threads_started":1}
I20260928 06:37:16.816767 2501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57ac801409aa4163aed7a410b139310e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:37:16.816922 2591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:16.812236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39238 (local address 127.16.157.66:32999)
0928 06:37:16.812507 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:16.812512 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:16.812523 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:37:16.812591 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:16.812592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:16.812609 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:16.812680 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:37:16.812884 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:16.814094 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:16.815185 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:16.816130 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:16.816209 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:16.816289 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:16.816535 (+ 246us) server_negotiation.cc:299] Negotiation successful
0928 06:37:16.816800 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":112,"threads_started":1}
I20260928 06:37:16.817292 2590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:16.812172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44842 (local address 127.16.157.65:35789)
0928 06:37:16.812407 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:16.812413 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:16.812473 (+ 60us) server_negotiation.cc:407] Connection header received
0928 06:37:16.812667 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:16.812669 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:16.812686 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:16.812740 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:37:16.812942 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:16.814098 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:16.816153 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:16.816956 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:16.817033 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:16.817114 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:16.817168 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 06:37:16.817210 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":134,"threads_started":1}
I20260928 06:37:16.817482 2586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:16.811874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.65:35789 (local address 127.0.0.1:44842)
0928 06:37:16.812411 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:16.812421 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:16.812500 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:16.812830 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:16.812831 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:16.812833 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:37:16.812879 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:16.812881 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:16.815165 (+ 2284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:16.815166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:16.816069 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:16.816071 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:16.817034 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:16.817035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:16.817091 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:16.817094 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:37:16.817373 (+ 279us) client_negotiation.cc:240] Negotiation successful
0928 06:37:16.817420 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"mutex_wait_us":5,"thread_start_us":87,"threads_started":1}
I20260928 06:37:16.817727 2381 tablet_service.cc:1467] Processing CreateTablet for tablet 57ac801409aa4163aed7a410b139310e (DEFAULT_TABLE table=test-table [id=3bae3d683f20448dbb47d42b094fab08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:37:16.818034 2381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57ac801409aa4163aed7a410b139310e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:37:16.818050 2250 tablet_service.cc:1467] Processing CreateTablet for tablet 57ac801409aa4163aed7a410b139310e (DEFAULT_TABLE table=test-table [id=3bae3d683f20448dbb47d42b094fab08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:37:16.818279 2250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57ac801409aa4163aed7a410b139310e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:37:16.820809 2593 tablet_bootstrap.cc:492] T 57ac801409aa4163aed7a410b139310e P 9efb92170945411286d0c19a07906918: Bootstrap starting.
I20260928 06:37:16.821293 2594 tablet_bootstrap.cc:492] T 57ac801409aa4163aed7a410b139310e P b372b04e434f448cab24f2eaad618642: Bootstrap starting.
I20260928 06:37:16.821516 2593 tablet_bootstrap.cc:654] T 57ac801409aa4163aed7a410b139310e P 9efb92170945411286d0c19a07906918: Neither blocks nor log segments found. Creating new log.
I20260928 06:37:16.821786 2592 tablet_bootstrap.cc:492] T 57ac801409aa4163aed7a410b139310e P 419b1305a17b48b5ac381396ee175e9c: Bootstrap starting.
I20260928 06:37:16.821913 2593 log.cc:824] T 57ac801409aa4163aed7a410b139310e P 9efb92170945411286d0c19a07906918: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:16.822027 2594 tablet_bootstrap.cc:654] T 57ac801409aa4163aed7a410b139310e P b372b04e434f448cab24f2eaad618642: Neither blocks nor log segments found. Creating new log.
I20260928 06:37:16.822388 2594 log.cc:824] T 57ac801409aa4163aed7a410b139310e P b372b04e434f448cab24f2eaad618642: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:16.822433 2592 tablet_bootstrap.cc:654] T 57ac801409aa4163aed7a410b139310e P 419b1305a17b48b5ac381396ee175e9c: Neither blocks nor log segments found. Creating new log.
I20260928 06:37:16.822727 2592 log.cc:824] T 57ac801409aa4163aed7a410b139310e P 419b1305a17b48b5ac381396ee175e9c: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:16.823463 2593 tablet_bootstrap.cc:492] T 57ac801409aa4163aed7a410b139310e P 9efb92170945411286d0c19a07906918: No bootstrap required, opened a new log
I20260928 06:37:16.823477 2592 tablet_bootstrap.cc:492] T 57ac801409aa4163aed7a410b139310e P 419b1305a17b48b5ac381396ee175e9c: No bootstrap required, opened a new log
I20260928 06:37:16.823542 2593 ts_tablet_manager.cc:1397] T 57ac801409aa4163aed7a410b139310e P 9efb92170945411286d0c19a07906918: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 06:37:16.823555 2592 ts_tablet_manager.cc:1397] T 57ac801409aa4163aed7a410b139310e P 419b1305a17b48b5ac381396ee175e9c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:37:16.823736 2594 tablet_bootstrap.cc:492] T 57ac801409aa4163aed7a410b139310e P b372b04e434f448cab24f2eaad618642: No bootstrap required, opened a new log
I20260928 06:37:16.823803 2594 ts_tablet_manager.cc:1397] T 57ac801409aa4163aed7a410b139310e P b372b04e434f448cab24f2eaad618642: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260928 06:37:16.825554 2592 raft_consensus.cc:348] T 57ac801409aa4163aed7a410b139310e P 419b1305a17b48b5ac381396ee175e9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efb92170945411286d0c19a07906918" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 35789 } } peers { permanent_uuid: "b372b04e434f448cab24f2eaad618642" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32999 } } peers { permanent_uuid: "419b1305a17b48b5ac381396ee175e9c" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39561 } }
I20260928 06:37:16.825784 2592 raft_consensus.cc:374] T 57ac801409aa4163aed7a410b139310e P 419b1305a17b48b5ac381396ee175e9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:37:16.825831 2592 raft_consensus.cc:729] T 57ac801409aa4163aed7a410b139310e P 419b1305a17b48b5ac381396ee175e9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 419b1305a17b48b5ac381396ee175e9c, State: Initialized, Role: FOLLOWER
I20260928 06:37:16.825941 2592 consensus_queue.cc:260] T 57ac801409aa4163aed7a410b139310e P 419b1305a17b48b5ac381396ee175e9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efb92170945411286d0c19a07906918" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 35789 } } peers { permanent_uuid: "b372b04e434f448cab24f2eaad618642" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32999 } } peers { permanent_uuid: "419b1305a17b48b5ac381396ee175e9c" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39561 } }
I20260928 06:37:16.826222 2592 ts_tablet_manager.cc:1428] T 57ac801409aa4163aed7a410b139310e P 419b1305a17b48b5ac381396ee175e9c: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 06:37:16.826243 2578 heartbeater.cc:503] Master 127.16.157.126:34053 was elected leader, sending a full tablet report...
I20260928 06:37:16.826373 2594 raft_consensus.cc:348] T 57ac801409aa4163aed7a410b139310e P b372b04e434f448cab24f2eaad618642 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efb92170945411286d0c19a07906918" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 35789 } } peers { permanent_uuid: "b372b04e434f448cab24f2eaad618642" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32999 } } peers { permanent_uuid: "419b1305a17b48b5ac381396ee175e9c" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39561 } }
I20260928 06:37:16.826373 2593 raft_consensus.cc:348] T 57ac801409aa4163aed7a410b139310e P 9efb92170945411286d0c19a07906918 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efb92170945411286d0c19a07906918" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 35789 } } peers { permanent_uuid: "b372b04e434f448cab24f2eaad618642" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32999 } } peers { permanent_uuid: "419b1305a17b48b5ac381396ee175e9c" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39561 } }
I20260928 06:37:16.826562 2594 raft_consensus.cc:374] T 57ac801409aa4163aed7a410b139310e P b372b04e434f448cab24f2eaad618642 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:37:16.826562 2593 raft_consensus.cc:374] T 57ac801409aa4163aed7a410b139310e P 9efb92170945411286d0c19a07906918 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:37:16.826612 2594 raft_consensus.cc:729] T 57ac801409aa4163aed7a410b139310e P b372b04e434f448cab24f2eaad618642 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b372b04e434f448cab24f2eaad618642, State: Initialized, Role: FOLLOWER
I20260928 06:37:16.826612 2593 raft_consensus.cc:729] T 57ac801409aa4163aed7a410b139310e P 9efb92170945411286d0c19a07906918 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9efb92170945411286d0c19a07906918, State: Initialized, Role: FOLLOWER
I20260928 06:37:16.826722 2593 consensus_queue.cc:260] T 57ac801409aa4163aed7a410b139310e P 9efb92170945411286d0c19a07906918 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efb92170945411286d0c19a07906918" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 35789 } } peers { permanent_uuid: "b372b04e434f448cab24f2eaad618642" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32999 } } peers { permanent_uuid: "419b1305a17b48b5ac381396ee175e9c" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39561 } }
I20260928 06:37:16.826722 2594 consensus_queue.cc:260] T 57ac801409aa4163aed7a410b139310e P b372b04e434f448cab24f2eaad618642 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efb92170945411286d0c19a07906918" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 35789 } } peers { permanent_uuid: "b372b04e434f448cab24f2eaad618642" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32999 } } peers { permanent_uuid: "419b1305a17b48b5ac381396ee175e9c" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39561 } }
I20260928 06:37:16.826982 2593 ts_tablet_manager.cc:1428] T 57ac801409aa4163aed7a410b139310e P 9efb92170945411286d0c19a07906918: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 06:37:16.826982 2594 ts_tablet_manager.cc:1428] T 57ac801409aa4163aed7a410b139310e P b372b04e434f448cab24f2eaad618642: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260928 06:37:16.827095 2447 heartbeater.cc:503] Master 127.16.157.126:34053 was elected leader, sending a full tablet report...
I20260928 06:37:16.827356 2316 heartbeater.cc:503] Master 127.16.157.126:34053 was elected leader, sending a full tablet report...
W20260928 06:37:16.917384 2448 tablet.cc:2369] T 57ac801409aa4163aed7a410b139310e P b372b04e434f448cab24f2eaad618642: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:37:17.025368 2317 tablet.cc:2369] T 57ac801409aa4163aed7a410b139310e P 9efb92170945411286d0c19a07906918: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:37:17.044632 2579 tablet.cc:2369] T 57ac801409aa4163aed7a410b139310e P 419b1305a17b48b5ac381396ee175e9c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:37:17.082727 2598 raft_consensus.cc:482] T 57ac801409aa4163aed7a410b139310e P 419b1305a17b48b5ac381396ee175e9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:37:17.082863 2598 raft_consensus.cc:504] T 57ac801409aa4163aed7a410b139310e P 419b1305a17b48b5ac381396ee175e9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efb92170945411286d0c19a07906918" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 35789 } } peers { permanent_uuid: "b372b04e434f448cab24f2eaad618642" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32999 } } peers { permanent_uuid: "419b1305a17b48b5ac381396ee175e9c" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39561 } }
I20260928 06:37:17.083176 2598 leader_election.cc:288] T 57ac801409aa4163aed7a410b139310e P 419b1305a17b48b5ac381396ee175e9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9efb92170945411286d0c19a07906918 (127.16.157.65:35789), b372b04e434f448cab24f2eaad618642 (127.16.157.66:32999)
I20260928 06:37:17.087296 2590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:17.083519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:45971 (local address 127.16.157.65:35789)
0928 06:37:17.083575 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:17.083578 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:17.083616 (+ 38us) server_negotiation.cc:407] Connection header received
0928 06:37:17.083649 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:17.083650 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:17.083663 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:17.083722 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:37:17.083938 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:17.084711 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:17.086329 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:17.086900 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:17.086981 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:17.086989 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:17.087174 (+ 185us) server_negotiation.cc:299] Negotiation successful
0928 06:37:17.087213 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 06:37:17.087287 2591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:17.083627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:42637 (local address 127.16.157.66:32999)
0928 06:37:17.083707 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:17.083709 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:17.083723 (+ 14us) server_negotiation.cc:407] Connection header received
0928 06:37:17.083782 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:17.083783 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:17.083808 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:17.083851 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:37:17.084018 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:17.084711 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:17.086322 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:17.086900 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:17.086981 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:17.086989 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:17.087174 (+ 185us) server_negotiation.cc:299] Negotiation successful
0928 06:37:17.087216 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 06:37:17.087285 2580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:17.083397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.65:35789 (local address 127.16.157.67:45971)
0928 06:37:17.083468 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:17.083481 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:17.083569 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:17.083746 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:17.083747 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:17.083749 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:37:17.083829 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:17.083831 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:17.084807 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:17.084808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:17.086221 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:17.086225 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:17.086999 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:17.087000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:17.087082 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:17.087087 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:37:17.087146 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 06:37:17.087197 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":49}
I20260928 06:37:17.087291 2601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:17.083517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.66:32999 (local address 127.16.157.67:42637)
0928 06:37:17.083682 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:17.083690 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:17.083730 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:17.083878 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:17.083879 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:17.083880 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:37:17.083943 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:17.083944 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:17.084807 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:17.084808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:17.086229 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:17.086230 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:17.087002 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:17.087003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:17.087091 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:17.087093 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:37:17.087146 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 06:37:17.087195 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":64,"thread_start_us":115,"threads_started":1}
I20260928 06:37:17.087742 2270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57ac801409aa4163aed7a410b139310e" candidate_uuid: "419b1305a17b48b5ac381396ee175e9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9efb92170945411286d0c19a07906918" is_pre_election: true
I20260928 06:37:17.087786 2401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57ac801409aa4163aed7a410b139310e" candidate_uuid: "419b1305a17b48b5ac381396ee175e9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b372b04e434f448cab24f2eaad618642" is_pre_election: true
I20260928 06:37:17.087917 2270 raft_consensus.cc:2454] T 57ac801409aa4163aed7a410b139310e P 9efb92170945411286d0c19a07906918 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 419b1305a17b48b5ac381396ee175e9c in term 0.
I20260928 06:37:17.087939 2401 raft_consensus.cc:2454] T 57ac801409aa4163aed7a410b139310e P b372b04e434f448cab24f2eaad618642 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 419b1305a17b48b5ac381396ee175e9c in term 0.
I20260928 06:37:17.088212 2465 leader_election.cc:302] T 57ac801409aa4163aed7a410b139310e P 419b1305a17b48b5ac381396ee175e9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 419b1305a17b48b5ac381396ee175e9c, b372b04e434f448cab24f2eaad618642; no voters:
I20260928 06:37:17.088356 2598 raft_consensus.cc:2781] T 57ac801409aa4163aed7a410b139310e P 419b1305a17b48b5ac381396ee175e9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:37:17.088444 2598 raft_consensus.cc:482] T 57ac801409aa4163aed7a410b139310e P 419b1305a17b48b5ac381396ee175e9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:37:17.088477 2598 raft_consensus.cc:3037] T 57ac801409aa4163aed7a410b139310e P 419b1305a17b48b5ac381396ee175e9c [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:37:17.089195 2598 raft_consensus.cc:504] T 57ac801409aa4163aed7a410b139310e P 419b1305a17b48b5ac381396ee175e9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efb92170945411286d0c19a07906918" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 35789 } } peers { permanent_uuid: "b372b04e434f448cab24f2eaad618642" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32999 } } peers { permanent_uuid: "419b1305a17b48b5ac381396ee175e9c" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39561 } }
I20260928 06:37:17.089346 2598 leader_election.cc:288] T 57ac801409aa4163aed7a410b139310e P 419b1305a17b48b5ac381396ee175e9c [CANDIDATE]: Term 1 election: Requested vote from peers 9efb92170945411286d0c19a07906918 (127.16.157.65:35789), b372b04e434f448cab24f2eaad618642 (127.16.157.66:32999)
I20260928 06:37:17.089524 2270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57ac801409aa4163aed7a410b139310e" candidate_uuid: "419b1305a17b48b5ac381396ee175e9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9efb92170945411286d0c19a07906918"
I20260928 06:37:17.089556 2401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57ac801409aa4163aed7a410b139310e" candidate_uuid: "419b1305a17b48b5ac381396ee175e9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b372b04e434f448cab24f2eaad618642"
I20260928 06:37:17.089640 2270 raft_consensus.cc:3037] T 57ac801409aa4163aed7a410b139310e P 9efb92170945411286d0c19a07906918 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:37:17.089653 2401 raft_consensus.cc:3037] T 57ac801409aa4163aed7a410b139310e P b372b04e434f448cab24f2eaad618642 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:37:17.090566 2401 raft_consensus.cc:2454] T 57ac801409aa4163aed7a410b139310e P b372b04e434f448cab24f2eaad618642 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 419b1305a17b48b5ac381396ee175e9c in term 1.
I20260928 06:37:17.090565 2270 raft_consensus.cc:2454] T 57ac801409aa4163aed7a410b139310e P 9efb92170945411286d0c19a07906918 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 419b1305a17b48b5ac381396ee175e9c in term 1.
I20260928 06:37:17.090766 2465 leader_election.cc:302] T 57ac801409aa4163aed7a410b139310e P 419b1305a17b48b5ac381396ee175e9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 419b1305a17b48b5ac381396ee175e9c, 9efb92170945411286d0c19a07906918; no voters:
I20260928 06:37:17.090857 2598 raft_consensus.cc:2781] T 57ac801409aa4163aed7a410b139310e P 419b1305a17b48b5ac381396ee175e9c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:37:17.091002 2598 raft_consensus.cc:686] T 57ac801409aa4163aed7a410b139310e P 419b1305a17b48b5ac381396ee175e9c [term 1 LEADER]: Becoming Leader. State: Replica: 419b1305a17b48b5ac381396ee175e9c, State: Running, Role: LEADER
I20260928 06:37:17.091099 2598 consensus_queue.cc:237] T 57ac801409aa4163aed7a410b139310e P 419b1305a17b48b5ac381396ee175e9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efb92170945411286d0c19a07906918" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 35789 } } peers { permanent_uuid: "b372b04e434f448cab24f2eaad618642" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32999 } } peers { permanent_uuid: "419b1305a17b48b5ac381396ee175e9c" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39561 } }
I20260928 06:37:17.091920 2128 catalog_manager.cc:5495] T 57ac801409aa4163aed7a410b139310e P 419b1305a17b48b5ac381396ee175e9c reported cstate change: term changed from 0 to 1, leader changed from <none> to 419b1305a17b48b5ac381396ee175e9c (127.16.157.67). New cstate: current_term: 1 leader_uuid: "419b1305a17b48b5ac381396ee175e9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efb92170945411286d0c19a07906918" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 35789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b372b04e434f448cab24f2eaad618642" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "419b1305a17b48b5ac381396ee175e9c" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 39561 } health_report { overall_health: HEALTHY } } }
I20260928 06:37:17.098424 2588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:17.096006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37572 (local address 127.16.157.67:39561)
0928 06:37:17.096076 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:17.096078 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:17.096087 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:37:17.096121 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:17.096121 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:17.096139 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:17.096227 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:37:17.096352 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:17.096826 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:17.097597 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:17.097804 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:17.097892 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:17.098066 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:37:17.098167 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:37:17.098254 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:17.098303 (+ 49us) server_negotiation.cc:299] Negotiation successful
0928 06:37:17.098323 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 06:37:17.100062 2401 raft_consensus.cc:1261] T 57ac801409aa4163aed7a410b139310e P b372b04e434f448cab24f2eaad618642 [term 1 FOLLOWER]: Refusing update from remote peer 419b1305a17b48b5ac381396ee175e9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:37:17.100086 2270 raft_consensus.cc:1261] T 57ac801409aa4163aed7a410b139310e P 9efb92170945411286d0c19a07906918 [term 1 FOLLOWER]: Refusing update from remote peer 419b1305a17b48b5ac381396ee175e9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:37:17.100459 2602 consensus_queue.cc:1035] T 57ac801409aa4163aed7a410b139310e P 419b1305a17b48b5ac381396ee175e9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b372b04e434f448cab24f2eaad618642" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 32999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:37:17.100591 2598 consensus_queue.cc:1035] T 57ac801409aa4163aed7a410b139310e P 419b1305a17b48b5ac381396ee175e9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9efb92170945411286d0c19a07906918" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:37:17.102666 2605 mvcc.cc:205] Tried to move back new op lower bound from 7334205182359453696 to 7334205182325661696. Current Snapshot: MvccSnapshot[applied={T|T < 7334205182359453696}]
I20260928 06:37:17.103259 2607 mvcc.cc:205] Tried to move back new op lower bound from 7334205182359453696 to 7334205182325661696. Current Snapshot: MvccSnapshot[applied={T|T < 7334205182359453696}]
I20260928 06:37:17.103317 2606 mvcc.cc:205] Tried to move back new op lower bound from 7334205182359453696 to 7334205182325661696. Current Snapshot: MvccSnapshot[applied={T|T < 7334205182359453696}]
I20260928 06:37:17.105149 2128 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:37:17.105243 2128 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:37:17.105991 2128 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a22cbe765eba4705b1b5c271e94b79af: Sending DeleteTablet for 3 replicas of tablet 57ac801409aa4163aed7a410b139310e
I20260928 06:37:17.106290 2250 tablet_service.cc:1514] Processing DeleteTablet for tablet 57ac801409aa4163aed7a410b139310e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:37:17 UTC) from {username='slave'} at 127.0.0.1:44842
I20260928 06:37:17.106362 2381 tablet_service.cc:1514] Processing DeleteTablet for tablet 57ac801409aa4163aed7a410b139310e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:37:17 UTC) from {username='slave'} at 127.0.0.1:39238
I20260928 06:37:17.106338 2501 tablet_service.cc:1514] Processing DeleteTablet for tablet 57ac801409aa4163aed7a410b139310e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:37:17 UTC) from {username='slave'} at 127.0.0.1:37560
I20260928 06:37:17.106532 2614 tablet_replica.cc:330] T 57ac801409aa4163aed7a410b139310e P 9efb92170945411286d0c19a07906918: stopping tablet replica
I20260928 06:37:17.106621 2616 tablet_replica.cc:330] T 57ac801409aa4163aed7a410b139310e P 419b1305a17b48b5ac381396ee175e9c: stopping tablet replica
I20260928 06:37:17.106631 2614 raft_consensus.cc:2229] T 57ac801409aa4163aed7a410b139310e P 9efb92170945411286d0c19a07906918 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:37:17.106742 2614 raft_consensus.cc:2258] T 57ac801409aa4163aed7a410b139310e P 9efb92170945411286d0c19a07906918 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:37:17.106822 2616 raft_consensus.cc:2229] T 57ac801409aa4163aed7a410b139310e P 419b1305a17b48b5ac381396ee175e9c [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:37:17.106889 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 2188
I20260928 06:37:17.107007 2616 raft_consensus.cc:2258] T 57ac801409aa4163aed7a410b139310e P 419b1305a17b48b5ac381396ee175e9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:37:17.107810 2616 ts_tablet_manager.cc:1905] T 57ac801409aa4163aed7a410b139310e P 419b1305a17b48b5ac381396ee175e9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:37:17.109864 2616 ts_tablet_manager.cc:1918] T 57ac801409aa4163aed7a410b139310e P 419b1305a17b48b5ac381396ee175e9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:37:17.109942 2616 log.cc:1196] T 57ac801409aa4163aed7a410b139310e P 419b1305a17b48b5ac381396ee175e9c: Deleting WAL directory at /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577196946045-17013-0/minicluster-data/ts-2/wal/wals/57ac801409aa4163aed7a410b139310e
I20260928 06:37:17.110191 2616 ts_tablet_manager.cc:1939] T 57ac801409aa4163aed7a410b139310e P 419b1305a17b48b5ac381396ee175e9c: Deleting consensus metadata
I20260928 06:37:17.110610 2113 catalog_manager.cc:4841] TS 419b1305a17b48b5ac381396ee175e9c (127.16.157.67:39561): tablet 57ac801409aa4163aed7a410b139310e (table test-table [id=3bae3d683f20448dbb47d42b094fab08]) successfully deleted
W20260928 06:37:17.112680 2112 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.157.65:35789 (error 108)
I20260928 06:37:17.114161 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 2319
I20260928 06:37:17.114167 2615 tablet_replica.cc:330] T 57ac801409aa4163aed7a410b139310e P b372b04e434f448cab24f2eaad618642: stopping tablet replica
I20260928 06:37:17.114289 2615 raft_consensus.cc:2229] T 57ac801409aa4163aed7a410b139310e P b372b04e434f448cab24f2eaad618642 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:37:17.114401 2615 raft_consensus.cc:2258] T 57ac801409aa4163aed7a410b139310e P b372b04e434f448cab24f2eaad618642 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:37:17.118413 2586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:17.118283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.65:35789 (local address 127.0.0.1:44850)
0928 06:37:17.118331 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:17.118373 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.65:35789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 06:37:17.118520 2112 catalog_manager.cc:4584] TS 9efb92170945411286d0c19a07906918 (127.16.157.65:35789): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 57ac801409aa4163aed7a410b139310e: Network error: Client connection negotiation failed: client connection to 127.16.157.65:35789: connect: Connection refused (error 111)
I20260928 06:37:17.120263 2586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:17.120159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.66:32999 (local address 127.0.0.1:39246)
0928 06:37:17.120210 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:17.120236 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.66:32999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 06:37:17.120340 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 2450
I20260928 06:37:17.125332 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 2097
2026-09-28T06:37:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (889 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:37:17 dist-test-slave-rrj5 krb5kdc[2620](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:37:17 dist-test-slave-rrj5 krb5kdc[2620](info): set up 2 sockets
Sep 28 06:37:17 dist-test-slave-rrj5 krb5kdc[2620](info): commencing operation
krb5kdc: starting...
W20260928 06:37:18.158174 17013 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Sep 28 06:37:18 dist-test-slave-rrj5 krb5kdc[2620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:37:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:37:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.157.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab.
Entry for principal kudu/127.16.157.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab.
I20260928 06:37:18.276891 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:37901
--webserver_interface=127.16.157.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=kudu/127.16.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:33003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.157.126:37901
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:37:18.355564 2633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:18.355757 2633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:18.355778 2633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:18.356838 2633 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:37:18.356869 2633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:18.356884 2633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:18.356895 2633 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:37:18.356907 2633 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:37:18.358392 2633 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:33003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.157.126:37901
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=kudu/127.16.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:37901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:37:18.358577 2633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:18.358741 2633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:18.363417 2639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:18.363395 2638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:18.363405 2641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:18.363687 2633 server_base.cc:1034] running on GCE node
I20260928 06:37:18.363907 2633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:18.364181 2633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:18.365326 2633 hybrid_clock.cc:648] HybridClock initialized: now 1790577438365315 us; error 31 us; skew 500 ppm
Sep 28 06:37:18 dist-test-slave-rrj5 krb5kdc[2620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577438, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:18.367611 2633 init.cc:377] Logged in from keytab as kudu/127.16.157.126@KRBTEST.COM (short username kudu)
I20260928 06:37:18.368038 2633 webserver.cc:460] Webserver started at http://127.16.157.126:33287/ using document root <none> and password file <none>
I20260928 06:37:18.368248 2633 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:18.368294 2633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:18.368435 2633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:37:18.369417 2633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/minicluster-data/master-0/data/instance:
uuid: "f7d04c5c48f347fe8a8309e4810a6015"
format_stamp: "Formatted at 2026-09-28 06:37:18 on dist-test-slave-rrj5"
server_key: "dede5fb804bf281582be6d4e21368d14"
server_key_iv: "d0a440d0ff25d74eb385e35bef95b77b"
server_key_version: "encryptionkey@0"
I20260928 06:37:18.369789 2633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/minicluster-data/master-0/wal/instance:
uuid: "f7d04c5c48f347fe8a8309e4810a6015"
format_stamp: "Formatted at 2026-09-28 06:37:18 on dist-test-slave-rrj5"
server_key: "dede5fb804bf281582be6d4e21368d14"
server_key_iv: "d0a440d0ff25d74eb385e35bef95b77b"
server_key_version: "encryptionkey@0"
I20260928 06:37:18.371122 2633 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:37:18.371840 2648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:18.372000 2633 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:37:18.372112 2633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/minicluster-data/master-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/minicluster-data/master-0/wal
uuid: "f7d04c5c48f347fe8a8309e4810a6015"
format_stamp: "Formatted at 2026-09-28 06:37:18 on dist-test-slave-rrj5"
server_key: "dede5fb804bf281582be6d4e21368d14"
server_key_iv: "d0a440d0ff25d74eb385e35bef95b77b"
server_key_version: "encryptionkey@0"
I20260928 06:37:18.372180 2633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:18.391103 2633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:18.392961 2633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:18.393119 2633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:18.397122 2633 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.126:37901
I20260928 06:37:18.397159 2700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.126:37901 every 8 connection(s)
I20260928 06:37:18.397480 2633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
I20260928 06:37:18.398303 2701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:37:18.400457 2701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7d04c5c48f347fe8a8309e4810a6015: Bootstrap starting.
I20260928 06:37:18.401039 2701 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7d04c5c48f347fe8a8309e4810a6015: Neither blocks nor log segments found. Creating new log.
I20260928 06:37:18.401335 2701 log.cc:824] T 00000000000000000000000000000000 P f7d04c5c48f347fe8a8309e4810a6015: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:18.401742 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 2633
I20260928 06:37:18.401875 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/minicluster-data/master-0/wal/instance
I20260928 06:37:18.402066 17013 external_mini_cluster.cc:1428] Setting key f4f475922e95023fa89447640b1ca73e
I20260928 06:37:18.402179 2701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7d04c5c48f347fe8a8309e4810a6015: No bootstrap required, opened a new log
I20260928 06:37:18.404126 2701 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7d04c5c48f347fe8a8309e4810a6015 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7d04c5c48f347fe8a8309e4810a6015" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 37901 } }
I20260928 06:37:18.404286 2701 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7d04c5c48f347fe8a8309e4810a6015 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:37:18.404337 2701 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7d04c5c48f347fe8a8309e4810a6015 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7d04c5c48f347fe8a8309e4810a6015, State: Initialized, Role: FOLLOWER
I20260928 06:37:18.404464 2701 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7d04c5c48f347fe8a8309e4810a6015 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7d04c5c48f347fe8a8309e4810a6015" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 37901 } }
I20260928 06:37:18.404539 2701 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7d04c5c48f347fe8a8309e4810a6015 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:37:18.404572 2701 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7d04c5c48f347fe8a8309e4810a6015 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:37:18.404623 2701 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7d04c5c48f347fe8a8309e4810a6015 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:37:18.405164 2701 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7d04c5c48f347fe8a8309e4810a6015 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7d04c5c48f347fe8a8309e4810a6015" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 37901 } }
I20260928 06:37:18.405309 2701 leader_election.cc:302] T 00000000000000000000000000000000 P f7d04c5c48f347fe8a8309e4810a6015 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7d04c5c48f347fe8a8309e4810a6015; no voters:
I20260928 06:37:18.405498 2701 leader_election.cc:288] T 00000000000000000000000000000000 P f7d04c5c48f347fe8a8309e4810a6015 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:37:18.405527 2706 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7d04c5c48f347fe8a8309e4810a6015 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:37:18.405663 2706 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7d04c5c48f347fe8a8309e4810a6015 [term 1 LEADER]: Becoming Leader. State: Replica: f7d04c5c48f347fe8a8309e4810a6015, State: Running, Role: LEADER
I20260928 06:37:18.405790 2701 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7d04c5c48f347fe8a8309e4810a6015 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:37:18.405757 2706 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7d04c5c48f347fe8a8309e4810a6015 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7d04c5c48f347fe8a8309e4810a6015" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 37901 } }
I20260928 06:37:18.406100 2708 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7d04c5c48f347fe8a8309e4810a6015 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7d04c5c48f347fe8a8309e4810a6015" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7d04c5c48f347fe8a8309e4810a6015" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 37901 } } }
I20260928 06:37:18.406195 2708 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7d04c5c48f347fe8a8309e4810a6015 [sys.catalog]: This master's current role is: LEADER
I20260928 06:37:18.406370 2707 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7d04c5c48f347fe8a8309e4810a6015 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7d04c5c48f347fe8a8309e4810a6015. Latest consensus state: current_term: 1 leader_uuid: "f7d04c5c48f347fe8a8309e4810a6015" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7d04c5c48f347fe8a8309e4810a6015" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 37901 } } }
I20260928 06:37:18.406428 2707 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7d04c5c48f347fe8a8309e4810a6015 [sys.catalog]: This master's current role is: LEADER
I20260928 06:37:18.406527 2712 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:37:18.406998 2712 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:37:18.408898 2712 catalog_manager.cc:1347] Generated new cluster ID: 0436c72647144357912e6c0d0ceacba3
I20260928 06:37:18.408957 2712 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:37:18 dist-test-slave-rrj5 krb5kdc[2620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:37:18.414167 2705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:18.402507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49448 (local address 127.16.157.126:37901)
0928 06:37:18.402756 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:18.402763 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:18.402791 (+ 28us) server_negotiation.cc:407] Connection header received
0928 06:37:18.403463 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:18.403470 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:18.403514 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:18.403658 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:18.404304 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:18.405702 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:18.406808 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:18.407501 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:18.407942 (+ 441us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:18.410573 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:18.410577 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:18.410578 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:18.410589 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:18.412463 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:18.412865 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:18.412868 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:18.412868 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:18.412964 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:18.413175 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:18.413175 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:18.413176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:18.413435 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:18.413610 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:18.413770 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 06:37:18.413887 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":104,"threads_started":1}
I20260928 06:37:18.423811 2712 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:37:18.424295 2712 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:37:18.430229 2712 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7d04c5c48f347fe8a8309e4810a6015: Generated new TSK 0
I20260928 06:37:18.430404 2712 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.157.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab.
Entry for principal kudu/127.16.157.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab.
I20260928 06:37:18.456846 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.65:0
--local_ip_for_outbound_sockets=127.16.157.65
--webserver_interface=127.16.157.65
--webserver_port=0
--tserver_master_addrs=127.16.157.126:37901
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab
--principal=kudu/127.16.157.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:33003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:37:18.535454 2727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:18.535645 2727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:18.535668 2727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:18.536796 2727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:18.536836 2727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:18.536887 2727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.65
I20260928 06:37:18.538255 2727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:33003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab
--principal=kudu/127.16.157.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.157.65
--webserver_port=0
--tserver_master_addrs=127.16.157.126:37901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.65
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:37:18.538491 2727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:18.538700 2727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:18.543454 2735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:18.543478 2733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:18.543515 2727 server_base.cc:1034] running on GCE node
W20260928 06:37:18.543481 2732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:18.543852 2727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:18.544090 2727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:18.545245 2727 hybrid_clock.cc:648] HybridClock initialized: now 1790577438545229 us; error 36 us; skew 500 ppm
Sep 28 06:37:18 dist-test-slave-rrj5 krb5kdc[2620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577438, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:18.547668 2727 init.cc:377] Logged in from keytab as kudu/127.16.157.65@KRBTEST.COM (short username kudu)
I20260928 06:37:18.548062 2727 webserver.cc:460] Webserver started at http://127.16.157.65:36817/ using document root <none> and password file <none>
I20260928 06:37:18.548291 2727 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:18.548336 2727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:18.548496 2727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:37:18.549459 2727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/minicluster-data/ts-0/data/instance:
uuid: "59cb73ff0f3a40f78a70ba7d246eee0d"
format_stamp: "Formatted at 2026-09-28 06:37:18 on dist-test-slave-rrj5"
server_key: "5a3c2276e377a23b0b5bb7bce5f80615"
server_key_iv: "782c905dbd6df5f03980532ece4e5c9a"
server_key_version: "encryptionkey@0"
I20260928 06:37:18.549865 2727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/minicluster-data/ts-0/wal/instance:
uuid: "59cb73ff0f3a40f78a70ba7d246eee0d"
format_stamp: "Formatted at 2026-09-28 06:37:18 on dist-test-slave-rrj5"
server_key: "5a3c2276e377a23b0b5bb7bce5f80615"
server_key_iv: "782c905dbd6df5f03980532ece4e5c9a"
server_key_version: "encryptionkey@0"
I20260928 06:37:18.551136 2727 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:37:18.551820 2742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:18.552004 2727 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:37:18.552090 2727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/minicluster-data/ts-0/wal
uuid: "59cb73ff0f3a40f78a70ba7d246eee0d"
format_stamp: "Formatted at 2026-09-28 06:37:18 on dist-test-slave-rrj5"
server_key: "5a3c2276e377a23b0b5bb7bce5f80615"
server_key_iv: "782c905dbd6df5f03980532ece4e5c9a"
server_key_version: "encryptionkey@0"
I20260928 06:37:18.552173 2727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:18.570025 2727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:18.571777 2727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:18.571933 2727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:18.572145 2727 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:37:18.572436 2727 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:37:18.572470 2727 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:18.572490 2727 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:37:18.572537 2727 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:18.578650 2727 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.65:43235
I20260928 06:37:18.578727 2855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.65:43235 every 8 connection(s)
I20260928 06:37:18.579025 2727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
I20260928 06:37:18.581187 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 2727
I20260928 06:37:18.581332 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/minicluster-data/ts-0/wal/instance
I20260928 06:37:18.581621 17013 external_mini_cluster.cc:1428] Setting key 7016085cc95d881121719d96cfd22c3f
Sep 28 06:37:18 dist-test-slave-rrj5 krb5kdc[2620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577438, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.65@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:37:18.588929 2858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:18.579683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:37901 (local address 127.16.157.65:58739)
0928 06:37:18.579862 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:18.579896 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:18.580337 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:18.580532 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:18.580535 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:18.581112 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:37:18.581348 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:18.581355 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:18.582187 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:18.582188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:18.582992 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:18.582994 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:18.583474 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:18.583475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:18.583637 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:18.584121 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:37:18.584132 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:37:18.585757 (+ 1625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:37:18.587604 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:18.587632 (+ 28us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:18.587634 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:18.587848 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:18.588067 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:18.588068 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:18.588068 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:18.588165 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:18.588461 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:37:18.588466 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:37:18.588569 (+ 103us) client_negotiation.cc:769] Sending connection context
0928 06:37:18.588670 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 06:37:18.588783 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":93,"threads_started":1}
I20260928 06:37:18.588938 2705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:18.579738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:58739 (local address 127.16.157.126:37901)
0928 06:37:18.579817 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:18.579819 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:18.579962 (+ 143us) server_negotiation.cc:407] Connection header received
0928 06:37:18.580420 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:18.580421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:18.580431 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:18.580481 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:18.581452 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:18.582087 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:18.583088 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:18.583316 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:18.583407 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:18.585848 (+ 2441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:18.585850 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:18.585850 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:18.585859 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:18.587515 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:18.587927 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:18.587928 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:18.587929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:18.587998 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:18.588232 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:18.588233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:18.588234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:18.588392 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:18.588468 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:18.588723 (+ 255us) server_negotiation.cc:299] Negotiation successful
0928 06:37:18.588816 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 06:37:18.589603 2856 heartbeater.cc:347] Connected to a master server at 127.16.157.126:37901
I20260928 06:37:18.589704 2856 heartbeater.cc:464] Registering TS with master...
I20260928 06:37:18.589871 2856 heartbeater.cc:511] Master 127.16.157.126:37901 requested a full tablet report, sending...
I20260928 06:37:18.590413 2665 ts_manager.cc:194] Registered new tserver with Master: 59cb73ff0f3a40f78a70ba7d246eee0d (127.16.157.65:43235)
WARNING: no policy specified for kudu/127.16.157.66@KRBTEST.COM; defaulting to no policy
I20260928 06:37:18.591056 2665 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.157.65@KRBTEST.COM'} at 127.16.157.65:58739
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab.
Entry for principal kudu/127.16.157.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab.
I20260928 06:37:18.608366 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.66:0
--local_ip_for_outbound_sockets=127.16.157.66
--webserver_interface=127.16.157.66
--webserver_port=0
--tserver_master_addrs=127.16.157.126:37901
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab
--principal=kudu/127.16.157.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:33003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:37:18.687410 2861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:18.687608 2861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:18.687629 2861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:18.689311 2861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:18.689379 2861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:18.689437 2861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.66
I20260928 06:37:18.690940 2861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:33003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab
--principal=kudu/127.16.157.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.157.66
--webserver_port=0
--tserver_master_addrs=127.16.157.126:37901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2861
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.66
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:37:18.691169 2861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:18.691418 2861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:18.696389 2866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:18.696389 2867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:18.696421 2869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:18.696615 2861 server_base.cc:1034] running on GCE node
I20260928 06:37:18.696831 2861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:18.697047 2861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:18.698218 2861 hybrid_clock.cc:648] HybridClock initialized: now 1790577438698204 us; error 27 us; skew 500 ppm
Sep 28 06:37:18 dist-test-slave-rrj5 krb5kdc[2620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577438, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:18.700579 2861 init.cc:377] Logged in from keytab as kudu/127.16.157.66@KRBTEST.COM (short username kudu)
I20260928 06:37:18.701063 2861 webserver.cc:460] Webserver started at http://127.16.157.66:42559/ using document root <none> and password file <none>
I20260928 06:37:18.701287 2861 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:18.701356 2861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:18.701506 2861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:37:18.702562 2861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/minicluster-data/ts-1/data/instance:
uuid: "d71741fa823b4a28819d28d5af7af200"
format_stamp: "Formatted at 2026-09-28 06:37:18 on dist-test-slave-rrj5"
server_key: "45a876248fecc93e5b5851e139553b30"
server_key_iv: "8093d0c4480e21824240fd82a0ef9fb5"
server_key_version: "encryptionkey@0"
I20260928 06:37:18.702934 2861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/minicluster-data/ts-1/wal/instance:
uuid: "d71741fa823b4a28819d28d5af7af200"
format_stamp: "Formatted at 2026-09-28 06:37:18 on dist-test-slave-rrj5"
server_key: "45a876248fecc93e5b5851e139553b30"
server_key_iv: "8093d0c4480e21824240fd82a0ef9fb5"
server_key_version: "encryptionkey@0"
I20260928 06:37:18.704243 2861 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:37:18.704943 2876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:18.705210 2861 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:37:18.705322 2861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/minicluster-data/ts-1/wal
uuid: "d71741fa823b4a28819d28d5af7af200"
format_stamp: "Formatted at 2026-09-28 06:37:18 on dist-test-slave-rrj5"
server_key: "45a876248fecc93e5b5851e139553b30"
server_key_iv: "8093d0c4480e21824240fd82a0ef9fb5"
server_key_version: "encryptionkey@0"
I20260928 06:37:18.705404 2861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:18.720340 2861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:18.722076 2861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:18.722239 2861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:18.722481 2861 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:37:18.722795 2861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:37:18.722829 2861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:18.722851 2861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:37:18.722863 2861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:18.728945 2861 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.66:40297
I20260928 06:37:18.729007 2989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.66:40297 every 8 connection(s)
I20260928 06:37:18.729275 2861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
I20260928 06:37:18.732856 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 2861
I20260928 06:37:18.732971 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/minicluster-data/ts-1/wal/instance
I20260928 06:37:18.733153 17013 external_mini_cluster.cc:1428] Setting key 6f825c0ea5c6e31471727bcb137f111a
Sep 28 06:37:18 dist-test-slave-rrj5 krb5kdc[2620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577438, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.66@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:37:18.739045 2705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:18.729974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.66:48871 (local address 127.16.157.126:37901)
0928 06:37:18.730050 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:18.730052 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:18.730198 (+ 146us) server_negotiation.cc:407] Connection header received
0928 06:37:18.730638 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:18.730639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:18.730648 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:18.730704 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:18.731588 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:18.732090 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:18.733340 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:18.733635 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:18.733701 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:18.736168 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:18.736169 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:18.736169 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:18.736184 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:18.737675 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:18.738130 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:18.738130 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:18.738131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:18.738181 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:18.738447 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:18.738447 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:18.738448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:18.738541 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:18.738589 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:18.738927 (+ 338us) server_negotiation.cc:299] Negotiation successful
0928 06:37:18.738986 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 06:37:18.739356 2992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:18.729925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:37901 (local address 127.16.157.66:48871)
0928 06:37:18.730075 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:18.730109 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:18.730561 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:18.730755 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:18.730758 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:18.731321 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:37:18.731501 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:18.731506 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:18.732169 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:18.732169 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:18.733240 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:18.733244 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:18.733710 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:18.733711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:18.733848 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:18.734311 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:37:18.734320 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:37:18.736077 (+ 1757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:37:18.737737 (+ 1660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:18.737744 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:18.737746 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:18.738055 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:18.738240 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:18.738241 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:18.738241 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:18.738382 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:18.738608 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:37:18.738615 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:37:18.738785 (+ 170us) client_negotiation.cc:769] Sending connection context
0928 06:37:18.738909 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 06:37:18.739083 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":91,"threads_started":1}
I20260928 06:37:18.739836 2990 heartbeater.cc:347] Connected to a master server at 127.16.157.126:37901
I20260928 06:37:18.739934 2990 heartbeater.cc:464] Registering TS with master...
I20260928 06:37:18.740120 2990 heartbeater.cc:511] Master 127.16.157.126:37901 requested a full tablet report, sending...
I20260928 06:37:18.740494 2665 ts_manager.cc:194] Registered new tserver with Master: d71741fa823b4a28819d28d5af7af200 (127.16.157.66:40297)
I20260928 06:37:18.740844 2665 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.157.66@KRBTEST.COM'} at 127.16.157.66:48871
WARNING: no policy specified for kudu/127.16.157.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab.
Entry for principal kudu/127.16.157.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab.
I20260928 06:37:18.760409 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.67:0
--local_ip_for_outbound_sockets=127.16.157.67
--webserver_interface=127.16.157.67
--webserver_port=0
--tserver_master_addrs=127.16.157.126:37901
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab
--principal=kudu/127.16.157.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:33003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:37:18.840384 2995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:18.840567 2995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:18.840587 2995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:18.841657 2995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:18.841688 2995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:18.841713 2995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.67
I20260928 06:37:18.843003 2995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:33003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab
--principal=kudu/127.16.157.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.157.67
--webserver_port=0
--tserver_master_addrs=127.16.157.126:37901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.67
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:37:18.843158 2995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:18.843317 2995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:18.848271 3003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:18.848302 3001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:18.848306 3000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:18.848476 2995 server_base.cc:1034] running on GCE node
I20260928 06:37:18.848732 2995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:18.848928 2995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:18.850063 2995 hybrid_clock.cc:648] HybridClock initialized: now 1790577438850054 us; error 30 us; skew 500 ppm
Sep 28 06:37:18 dist-test-slave-rrj5 krb5kdc[2620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577438, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:18.852551 2995 init.cc:377] Logged in from keytab as kudu/127.16.157.67@KRBTEST.COM (short username kudu)
I20260928 06:37:18.852981 2995 webserver.cc:460] Webserver started at http://127.16.157.67:33611/ using document root <none> and password file <none>
I20260928 06:37:18.853197 2995 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:18.853241 2995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:18.853425 2995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:37:18.854498 2995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/minicluster-data/ts-2/data/instance:
uuid: "d0f3323eaa4b42f083730f7c2f294786"
format_stamp: "Formatted at 2026-09-28 06:37:18 on dist-test-slave-rrj5"
server_key: "f565913cca62c979fec15e1b939c0470"
server_key_iv: "4fd705a68dd88125e9400ac45b7b78ac"
server_key_version: "encryptionkey@0"
I20260928 06:37:18.854852 2995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/minicluster-data/ts-2/wal/instance:
uuid: "d0f3323eaa4b42f083730f7c2f294786"
format_stamp: "Formatted at 2026-09-28 06:37:18 on dist-test-slave-rrj5"
server_key: "f565913cca62c979fec15e1b939c0470"
server_key_iv: "4fd705a68dd88125e9400ac45b7b78ac"
server_key_version: "encryptionkey@0"
I20260928 06:37:18.856243 2995 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:37:18.856992 3010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:18.857177 2995 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:37:18.857292 2995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/minicluster-data/ts-2/wal
uuid: "d0f3323eaa4b42f083730f7c2f294786"
format_stamp: "Formatted at 2026-09-28 06:37:18 on dist-test-slave-rrj5"
server_key: "f565913cca62c979fec15e1b939c0470"
server_key_iv: "4fd705a68dd88125e9400ac45b7b78ac"
server_key_version: "encryptionkey@0"
I20260928 06:37:18.857372 2995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:18.870514 2995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:18.873965 2995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:18.874184 2995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:18.874459 2995 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:37:18.874842 2995 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:37:18.874888 2995 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:18.874918 2995 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:37:18.874955 2995 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:18.881098 2995 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.67:42127
I20260928 06:37:18.881196 3123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.67:42127 every 8 connection(s)
I20260928 06:37:18.881496 2995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
I20260928 06:37:18.884776 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 2995
I20260928 06:37:18.884866 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/minicluster-data/ts-2/wal/instance
I20260928 06:37:18.885084 17013 external_mini_cluster.cc:1428] Setting key df4fbb16e048e353d4eb7431b9b62e5a
Sep 28 06:37:18 dist-test-slave-rrj5 krb5kdc[2620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577438, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.67@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:37:18.890631 2705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:18.882260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:46519 (local address 127.16.157.126:37901)
0928 06:37:18.882321 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:18.882344 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:18.882580 (+ 236us) server_negotiation.cc:407] Connection header received
0928 06:37:18.883065 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:18.883066 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:18.883076 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:18.883128 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:18.884000 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:18.884498 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:18.885853 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:18.886131 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:18.886201 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:18.888277 (+ 2076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:18.888278 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:18.888279 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:18.888284 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:18.889449 (+ 1165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:18.889856 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:18.889857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:18.889857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:18.889909 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:18.890115 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:18.890115 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:18.890116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:18.890213 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:18.890263 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:18.890502 (+ 239us) server_negotiation.cc:299] Negotiation successful
0928 06:37:18.890569 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 06:37:18.890810 3126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:18.882255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:37901 (local address 127.16.157.67:46519)
0928 06:37:18.882467 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:18.882500 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:18.882984 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:18.883182 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:18.883186 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:18.883735 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:37:18.883926 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:18.883931 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:18.884574 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:18.884575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:18.885776 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:18.885779 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:18.886223 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:18.886224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:18.886326 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:18.886844 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:37:18.886853 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:37:18.888200 (+ 1347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:37:18.889520 (+ 1320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:18.889526 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:18.889528 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:18.889792 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:18.889968 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:18.889968 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:18.889968 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:18.890062 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:18.890267 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:37:18.890271 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:37:18.890396 (+ 125us) client_negotiation.cc:769] Sending connection context
0928 06:37:18.890476 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 06:37:18.890591 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":137,"threads_started":1}
I20260928 06:37:18.891283 3124 heartbeater.cc:347] Connected to a master server at 127.16.157.126:37901
I20260928 06:37:18.891371 3124 heartbeater.cc:464] Registering TS with master...
I20260928 06:37:18.891503 3124 heartbeater.cc:511] Master 127.16.157.126:37901 requested a full tablet report, sending...
I20260928 06:37:18.891829 2665 ts_manager.cc:194] Registered new tserver with Master: d0f3323eaa4b42f083730f7c2f294786 (127.16.157.67:42127)
I20260928 06:37:18.892155 2665 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.157.67@KRBTEST.COM'} at 127.16.157.67:46519
I20260928 06:37:18.896780 17013 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:37:18.903666 2705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:18.898129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49456 (local address 127.16.157.126:37901)
0928 06:37:18.898203 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:18.898205 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:18.898278 (+ 73us) server_negotiation.cc:407] Connection header received
0928 06:37:18.898321 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:18.898321 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:18.898329 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:18.898373 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:18.898861 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:18.899312 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:18.899992 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:18.900160 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:18.900216 (+ 56us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:18.901367 (+ 1151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:18.901368 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:18.901368 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:18.901375 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:18.902605 (+ 1230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:18.902975 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:18.902975 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:18.902976 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:18.903024 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:18.903218 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:18.903218 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:18.903219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:18.903332 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:18.903409 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:18.903514 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 06:37:18.903558 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 06:37:18.905254 2665 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49456:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:37:18.907078 2665 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:37:18.913439 3133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:18.909438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.67:42127 (local address 127.0.0.1:47178)
0928 06:37:18.909872 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:18.909883 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:18.909938 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:18.910199 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:18.910200 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:18.910201 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:37:18.910248 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:18.910249 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:18.911118 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:18.911120 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:18.912340 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:18.912342 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:18.913156 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:18.913157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:18.913233 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:18.913247 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 06:37:18.913320 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 06:37:18.913371 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":38,"threads_started":1}
I20260928 06:37:18.913591 3136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:18.909221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.65:43235 (local address 127.0.0.1:43926)
0928 06:37:18.909513 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:18.909524 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:18.909592 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:18.910043 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:18.910043 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:18.910047 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:37:18.910078 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:18.910079 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:18.911464 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:18.911465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:18.912483 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:18.912490 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:18.913341 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:18.913342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:18.913407 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:18.913411 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:37:18.913467 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 06:37:18.913506 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":52,"threads_started":1}
I20260928 06:37:18.913611 3134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:18.909328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43926 (local address 127.16.157.65:43235)
0928 06:37:18.909843 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:18.909847 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:18.909858 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:37:18.909895 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:18.909897 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:18.909954 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:18.910150 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:37:18.910229 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:18.911355 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:18.912579 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:18.913250 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:18.913339 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:18.913393 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:18.913480 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 06:37:18.913517 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":80,"threads_started":1}
I20260928 06:37:18.913786 3137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:18.909603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47178 (local address 127.16.157.67:42127)
0928 06:37:18.909773 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:18.909778 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:18.910048 (+ 270us) server_negotiation.cc:407] Connection header received
0928 06:37:18.910091 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:18.910093 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:18.910108 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:18.910200 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:37:18.910335 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:18.911007 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:18.912418 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:18.913067 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:18.913142 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:18.913194 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:18.913642 (+ 448us) server_negotiation.cc:299] Negotiation successful
0928 06:37:18.913686 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":104,"threads_started":1}
I20260928 06:37:18.914454 2790 tablet_service.cc:1467] Processing CreateTablet for tablet 0e92744ce8794d609be0b33a9240e372 (DEFAULT_TABLE table=test-table [id=bd04737a223c446186c11a565b892ab3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:37:18.914446 3058 tablet_service.cc:1467] Processing CreateTablet for tablet 0e92744ce8794d609be0b33a9240e372 (DEFAULT_TABLE table=test-table [id=bd04737a223c446186c11a565b892ab3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:37:18.914803 3058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e92744ce8794d609be0b33a9240e372. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:37:18.914804 2790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e92744ce8794d609be0b33a9240e372. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:37:18.914875 3132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:18.909218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.66:40297 (local address 127.0.0.1:48474)
0928 06:37:18.909420 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:18.909458 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:18.909530 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:18.909802 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:18.909804 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:18.909807 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:37:18.909878 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:18.909880 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:18.912573 (+ 2693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:18.912573 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:18.913249 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:18.913250 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:18.914623 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:18.914624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:18.914686 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:18.914690 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:37:18.914748 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 06:37:18.914790 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":70,"threads_started":1}
I20260928 06:37:18.914916 3135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:18.909332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48474 (local address 127.16.157.66:40297)
0928 06:37:18.909611 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:18.909616 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:18.909629 (+ 13us) server_negotiation.cc:407] Connection header received
0928 06:37:18.909668 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:18.909670 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:18.909684 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:18.909776 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:37:18.910281 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:18.911358 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:18.913345 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:18.914528 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:18.914610 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:18.914665 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:18.914784 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 06:37:18.914842 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":78,"threads_started":1}
I20260928 06:37:18.915699 2924 tablet_service.cc:1467] Processing CreateTablet for tablet 0e92744ce8794d609be0b33a9240e372 (DEFAULT_TABLE table=test-table [id=bd04737a223c446186c11a565b892ab3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:37:18.916008 2924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e92744ce8794d609be0b33a9240e372. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:37:18.917349 3138 tablet_bootstrap.cc:492] T 0e92744ce8794d609be0b33a9240e372 P d0f3323eaa4b42f083730f7c2f294786: Bootstrap starting.
I20260928 06:37:18.918015 3138 tablet_bootstrap.cc:654] T 0e92744ce8794d609be0b33a9240e372 P d0f3323eaa4b42f083730f7c2f294786: Neither blocks nor log segments found. Creating new log.
I20260928 06:37:18.918562 3138 log.cc:824] T 0e92744ce8794d609be0b33a9240e372 P d0f3323eaa4b42f083730f7c2f294786: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:18.919217 3138 tablet_bootstrap.cc:492] T 0e92744ce8794d609be0b33a9240e372 P d0f3323eaa4b42f083730f7c2f294786: No bootstrap required, opened a new log
I20260928 06:37:18.919296 3138 ts_tablet_manager.cc:1397] T 0e92744ce8794d609be0b33a9240e372 P d0f3323eaa4b42f083730f7c2f294786: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:37:18.919400 3141 tablet_bootstrap.cc:492] T 0e92744ce8794d609be0b33a9240e372 P d71741fa823b4a28819d28d5af7af200: Bootstrap starting.
I20260928 06:37:18.919642 3139 tablet_bootstrap.cc:492] T 0e92744ce8794d609be0b33a9240e372 P 59cb73ff0f3a40f78a70ba7d246eee0d: Bootstrap starting.
I20260928 06:37:18.920038 3141 tablet_bootstrap.cc:654] T 0e92744ce8794d609be0b33a9240e372 P d71741fa823b4a28819d28d5af7af200: Neither blocks nor log segments found. Creating new log.
I20260928 06:37:18.920279 3139 tablet_bootstrap.cc:654] T 0e92744ce8794d609be0b33a9240e372 P 59cb73ff0f3a40f78a70ba7d246eee0d: Neither blocks nor log segments found. Creating new log.
I20260928 06:37:18.920665 3141 log.cc:824] T 0e92744ce8794d609be0b33a9240e372 P d71741fa823b4a28819d28d5af7af200: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:18.920857 3139 log.cc:824] T 0e92744ce8794d609be0b33a9240e372 P 59cb73ff0f3a40f78a70ba7d246eee0d: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:18.921140 3138 raft_consensus.cc:348] T 0e92744ce8794d609be0b33a9240e372 P d0f3323eaa4b42f083730f7c2f294786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d71741fa823b4a28819d28d5af7af200" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 40297 } } peers { permanent_uuid: "59cb73ff0f3a40f78a70ba7d246eee0d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 43235 } } peers { permanent_uuid: "d0f3323eaa4b42f083730f7c2f294786" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 42127 } }
I20260928 06:37:18.921331 3138 raft_consensus.cc:374] T 0e92744ce8794d609be0b33a9240e372 P d0f3323eaa4b42f083730f7c2f294786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:37:18.921375 3138 raft_consensus.cc:729] T 0e92744ce8794d609be0b33a9240e372 P d0f3323eaa4b42f083730f7c2f294786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0f3323eaa4b42f083730f7c2f294786, State: Initialized, Role: FOLLOWER
I20260928 06:37:18.921491 3141 tablet_bootstrap.cc:492] T 0e92744ce8794d609be0b33a9240e372 P d71741fa823b4a28819d28d5af7af200: No bootstrap required, opened a new log
I20260928 06:37:18.921489 3138 consensus_queue.cc:260] T 0e92744ce8794d609be0b33a9240e372 P d0f3323eaa4b42f083730f7c2f294786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d71741fa823b4a28819d28d5af7af200" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 40297 } } peers { permanent_uuid: "59cb73ff0f3a40f78a70ba7d246eee0d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 43235 } } peers { permanent_uuid: "d0f3323eaa4b42f083730f7c2f294786" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 42127 } }
I20260928 06:37:18.921558 3141 ts_tablet_manager.cc:1397] T 0e92744ce8794d609be0b33a9240e372 P d71741fa823b4a28819d28d5af7af200: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:37:18.921777 3138 ts_tablet_manager.cc:1428] T 0e92744ce8794d609be0b33a9240e372 P d0f3323eaa4b42f083730f7c2f294786: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260928 06:37:18.921789 3124 heartbeater.cc:503] Master 127.16.157.126:37901 was elected leader, sending a full tablet report...
I20260928 06:37:18.921969 3139 tablet_bootstrap.cc:492] T 0e92744ce8794d609be0b33a9240e372 P 59cb73ff0f3a40f78a70ba7d246eee0d: No bootstrap required, opened a new log
I20260928 06:37:18.922048 3139 ts_tablet_manager.cc:1397] T 0e92744ce8794d609be0b33a9240e372 P 59cb73ff0f3a40f78a70ba7d246eee0d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:37:18.923424 3141 raft_consensus.cc:348] T 0e92744ce8794d609be0b33a9240e372 P d71741fa823b4a28819d28d5af7af200 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d71741fa823b4a28819d28d5af7af200" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 40297 } } peers { permanent_uuid: "59cb73ff0f3a40f78a70ba7d246eee0d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 43235 } } peers { permanent_uuid: "d0f3323eaa4b42f083730f7c2f294786" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 42127 } }
I20260928 06:37:18.923550 3141 raft_consensus.cc:374] T 0e92744ce8794d609be0b33a9240e372 P d71741fa823b4a28819d28d5af7af200 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:37:18.923573 3141 raft_consensus.cc:729] T 0e92744ce8794d609be0b33a9240e372 P d71741fa823b4a28819d28d5af7af200 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d71741fa823b4a28819d28d5af7af200, State: Initialized, Role: FOLLOWER
I20260928 06:37:18.923695 3141 consensus_queue.cc:260] T 0e92744ce8794d609be0b33a9240e372 P d71741fa823b4a28819d28d5af7af200 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d71741fa823b4a28819d28d5af7af200" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 40297 } } peers { permanent_uuid: "59cb73ff0f3a40f78a70ba7d246eee0d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 43235 } } peers { permanent_uuid: "d0f3323eaa4b42f083730f7c2f294786" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 42127 } }
I20260928 06:37:18.923781 3139 raft_consensus.cc:348] T 0e92744ce8794d609be0b33a9240e372 P 59cb73ff0f3a40f78a70ba7d246eee0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d71741fa823b4a28819d28d5af7af200" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 40297 } } peers { permanent_uuid: "59cb73ff0f3a40f78a70ba7d246eee0d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 43235 } } peers { permanent_uuid: "d0f3323eaa4b42f083730f7c2f294786" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 42127 } }
I20260928 06:37:18.923961 3139 raft_consensus.cc:374] T 0e92744ce8794d609be0b33a9240e372 P 59cb73ff0f3a40f78a70ba7d246eee0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:37:18.923985 3141 ts_tablet_manager.cc:1428] T 0e92744ce8794d609be0b33a9240e372 P d71741fa823b4a28819d28d5af7af200: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 06:37:18.924003 3139 raft_consensus.cc:729] T 0e92744ce8794d609be0b33a9240e372 P 59cb73ff0f3a40f78a70ba7d246eee0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59cb73ff0f3a40f78a70ba7d246eee0d, State: Initialized, Role: FOLLOWER
I20260928 06:37:18.923995 2990 heartbeater.cc:503] Master 127.16.157.126:37901 was elected leader, sending a full tablet report...
I20260928 06:37:18.924105 3139 consensus_queue.cc:260] T 0e92744ce8794d609be0b33a9240e372 P 59cb73ff0f3a40f78a70ba7d246eee0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d71741fa823b4a28819d28d5af7af200" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 40297 } } peers { permanent_uuid: "59cb73ff0f3a40f78a70ba7d246eee0d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 43235 } } peers { permanent_uuid: "d0f3323eaa4b42f083730f7c2f294786" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 42127 } }
I20260928 06:37:18.924376 3139 ts_tablet_manager.cc:1428] T 0e92744ce8794d609be0b33a9240e372 P 59cb73ff0f3a40f78a70ba7d246eee0d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:37:18.924394 2856 heartbeater.cc:503] Master 127.16.157.126:37901 was elected leader, sending a full tablet report...
W20260928 06:37:18.979962 2991 tablet.cc:2369] T 0e92744ce8794d609be0b33a9240e372 P d71741fa823b4a28819d28d5af7af200: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:37:19.013091 3144 raft_consensus.cc:482] T 0e92744ce8794d609be0b33a9240e372 P d0f3323eaa4b42f083730f7c2f294786 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:37:19.013239 3144 raft_consensus.cc:504] T 0e92744ce8794d609be0b33a9240e372 P d0f3323eaa4b42f083730f7c2f294786 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d71741fa823b4a28819d28d5af7af200" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 40297 } } peers { permanent_uuid: "59cb73ff0f3a40f78a70ba7d246eee0d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 43235 } } peers { permanent_uuid: "d0f3323eaa4b42f083730f7c2f294786" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 42127 } }
I20260928 06:37:19.013690 3144 leader_election.cc:288] T 0e92744ce8794d609be0b33a9240e372 P d0f3323eaa4b42f083730f7c2f294786 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d71741fa823b4a28819d28d5af7af200 (127.16.157.66:40297), 59cb73ff0f3a40f78a70ba7d246eee0d (127.16.157.65:43235)
I20260928 06:37:19.017691 3134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:19.013966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:50439 (local address 127.16.157.65:43235)
0928 06:37:19.014017 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:19.014019 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:19.014029 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:37:19.014061 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:19.014062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:19.014076 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:19.014167 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:37:19.014390 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:19.015130 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:19.016687 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:19.017284 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:19.017362 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:19.017370 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:19.017560 (+ 190us) server_negotiation.cc:299] Negotiation successful
0928 06:37:19.017621 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 06:37:19.017693 3135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:19.013986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:38365 (local address 127.16.157.66:40297)
0928 06:37:19.014065 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:19.014067 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:19.014166 (+ 99us) server_negotiation.cc:407] Connection header received
0928 06:37:19.014216 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:19.014217 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:19.014235 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:19.014280 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:37:19.014448 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:19.015130 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:19.016685 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:19.017276 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:19.017362 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:19.017370 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:19.017559 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 06:37:19.017634 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 06:37:19.017773 3126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:19.013857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.65:43235 (local address 127.16.157.67:50439)
0928 06:37:19.013907 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:19.013923 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:19.013983 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:19.014194 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:19.014196 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:19.014197 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:37:19.014312 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:19.014313 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:19.015229 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:19.015229 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:19.016592 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:19.016595 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:19.017378 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:19.017379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:19.017457 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:19.017459 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:37:19.017515 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 06:37:19.017661 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":184}
I20260928 06:37:19.017957 3147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:19.013857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.66:40297 (local address 127.16.157.67:38365)
0928 06:37:19.014066 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:19.014084 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:19.014129 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:19.014316 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:19.014316 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:19.014318 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:37:19.014379 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:19.014380 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:19.015229 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:19.015229 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:19.016592 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:19.016595 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:19.017378 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:19.017379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:19.017454 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:19.017458 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:37:19.017515 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 06:37:19.017660 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":111,"thread_start_us":138,"threads_started":1}
I20260928 06:37:19.018147 2810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e92744ce8794d609be0b33a9240e372" candidate_uuid: "d0f3323eaa4b42f083730f7c2f294786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59cb73ff0f3a40f78a70ba7d246eee0d" is_pre_election: true
I20260928 06:37:19.018313 2810 raft_consensus.cc:2454] T 0e92744ce8794d609be0b33a9240e372 P 59cb73ff0f3a40f78a70ba7d246eee0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0f3323eaa4b42f083730f7c2f294786 in term 0.
I20260928 06:37:19.018362 2944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e92744ce8794d609be0b33a9240e372" candidate_uuid: "d0f3323eaa4b42f083730f7c2f294786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d71741fa823b4a28819d28d5af7af200" is_pre_election: true
I20260928 06:37:19.018532 2944 raft_consensus.cc:2454] T 0e92744ce8794d609be0b33a9240e372 P d71741fa823b4a28819d28d5af7af200 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0f3323eaa4b42f083730f7c2f294786 in term 0.
I20260928 06:37:19.018590 3011 leader_election.cc:302] T 0e92744ce8794d609be0b33a9240e372 P d0f3323eaa4b42f083730f7c2f294786 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59cb73ff0f3a40f78a70ba7d246eee0d, d0f3323eaa4b42f083730f7c2f294786; no voters:
I20260928 06:37:19.018713 3144 raft_consensus.cc:2781] T 0e92744ce8794d609be0b33a9240e372 P d0f3323eaa4b42f083730f7c2f294786 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:37:19.018795 3144 raft_consensus.cc:482] T 0e92744ce8794d609be0b33a9240e372 P d0f3323eaa4b42f083730f7c2f294786 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:37:19.018811 3144 raft_consensus.cc:3037] T 0e92744ce8794d609be0b33a9240e372 P d0f3323eaa4b42f083730f7c2f294786 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:37:19.019515 3144 raft_consensus.cc:504] T 0e92744ce8794d609be0b33a9240e372 P d0f3323eaa4b42f083730f7c2f294786 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d71741fa823b4a28819d28d5af7af200" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 40297 } } peers { permanent_uuid: "59cb73ff0f3a40f78a70ba7d246eee0d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 43235 } } peers { permanent_uuid: "d0f3323eaa4b42f083730f7c2f294786" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 42127 } }
I20260928 06:37:19.019673 3144 leader_election.cc:288] T 0e92744ce8794d609be0b33a9240e372 P d0f3323eaa4b42f083730f7c2f294786 [CANDIDATE]: Term 1 election: Requested vote from peers d71741fa823b4a28819d28d5af7af200 (127.16.157.66:40297), 59cb73ff0f3a40f78a70ba7d246eee0d (127.16.157.65:43235)
I20260928 06:37:19.019852 2944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e92744ce8794d609be0b33a9240e372" candidate_uuid: "d0f3323eaa4b42f083730f7c2f294786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d71741fa823b4a28819d28d5af7af200"
I20260928 06:37:19.019918 2810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e92744ce8794d609be0b33a9240e372" candidate_uuid: "d0f3323eaa4b42f083730f7c2f294786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59cb73ff0f3a40f78a70ba7d246eee0d"
I20260928 06:37:19.019917 2944 raft_consensus.cc:3037] T 0e92744ce8794d609be0b33a9240e372 P d71741fa823b4a28819d28d5af7af200 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:37:19.019995 2810 raft_consensus.cc:3037] T 0e92744ce8794d609be0b33a9240e372 P 59cb73ff0f3a40f78a70ba7d246eee0d [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:37:19.020704 2944 raft_consensus.cc:2454] T 0e92744ce8794d609be0b33a9240e372 P d71741fa823b4a28819d28d5af7af200 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0f3323eaa4b42f083730f7c2f294786 in term 1.
I20260928 06:37:19.020821 2810 raft_consensus.cc:2454] T 0e92744ce8794d609be0b33a9240e372 P 59cb73ff0f3a40f78a70ba7d246eee0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0f3323eaa4b42f083730f7c2f294786 in term 1.
I20260928 06:37:19.020871 3012 leader_election.cc:302] T 0e92744ce8794d609be0b33a9240e372 P d0f3323eaa4b42f083730f7c2f294786 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0f3323eaa4b42f083730f7c2f294786, d71741fa823b4a28819d28d5af7af200; no voters:
I20260928 06:37:19.020993 3144 raft_consensus.cc:2781] T 0e92744ce8794d609be0b33a9240e372 P d0f3323eaa4b42f083730f7c2f294786 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:37:19.021179 3144 raft_consensus.cc:686] T 0e92744ce8794d609be0b33a9240e372 P d0f3323eaa4b42f083730f7c2f294786 [term 1 LEADER]: Becoming Leader. State: Replica: d0f3323eaa4b42f083730f7c2f294786, State: Running, Role: LEADER
I20260928 06:37:19.021299 3144 consensus_queue.cc:237] T 0e92744ce8794d609be0b33a9240e372 P d0f3323eaa4b42f083730f7c2f294786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d71741fa823b4a28819d28d5af7af200" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 40297 } } peers { permanent_uuid: "59cb73ff0f3a40f78a70ba7d246eee0d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 43235 } } peers { permanent_uuid: "d0f3323eaa4b42f083730f7c2f294786" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 42127 } }
I20260928 06:37:19.022112 2665 catalog_manager.cc:5495] T 0e92744ce8794d609be0b33a9240e372 P d0f3323eaa4b42f083730f7c2f294786 reported cstate change: term changed from 0 to 1, leader changed from <none> to d0f3323eaa4b42f083730f7c2f294786 (127.16.157.67). New cstate: current_term: 1 leader_uuid: "d0f3323eaa4b42f083730f7c2f294786" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d71741fa823b4a28819d28d5af7af200" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 40297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59cb73ff0f3a40f78a70ba7d246eee0d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 43235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0f3323eaa4b42f083730f7c2f294786" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 42127 } health_report { overall_health: HEALTHY } } }
I20260928 06:37:19.030975 3137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:19.028879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47192 (local address 127.16.157.67:42127)
0928 06:37:19.028936 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:19.028938 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:19.028949 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:37:19.028974 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:19.028975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:19.028988 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:19.029044 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:37:19.029128 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:19.029555 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:19.030335 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:19.030510 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:19.030559 (+ 49us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:19.030706 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:37:19.030784 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:37:19.030852 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:19.030899 (+ 47us) server_negotiation.cc:299] Negotiation successful
0928 06:37:19.030908 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 06:37:19.032545 2944 raft_consensus.cc:1261] T 0e92744ce8794d609be0b33a9240e372 P d71741fa823b4a28819d28d5af7af200 [term 1 FOLLOWER]: Refusing update from remote peer d0f3323eaa4b42f083730f7c2f294786: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:37:19.032773 2810 raft_consensus.cc:1261] T 0e92744ce8794d609be0b33a9240e372 P 59cb73ff0f3a40f78a70ba7d246eee0d [term 1 FOLLOWER]: Refusing update from remote peer d0f3323eaa4b42f083730f7c2f294786: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:37:19.032893 3148 consensus_queue.cc:1035] T 0e92744ce8794d609be0b33a9240e372 P d0f3323eaa4b42f083730f7c2f294786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d71741fa823b4a28819d28d5af7af200" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 40297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:37:19.033020 3144 consensus_queue.cc:1035] T 0e92744ce8794d609be0b33a9240e372 P d0f3323eaa4b42f083730f7c2f294786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59cb73ff0f3a40f78a70ba7d246eee0d" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 43235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:37:19.035748 3153 mvcc.cc:205] Tried to move back new op lower bound from 7334205190275121152 to 7334205190231830528. Current Snapshot: MvccSnapshot[applied={T|T < 7334205190275121152}]
I20260928 06:37:19.037360 2663 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:37:19.037454 2663 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:37:19.038244 2663 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f7d04c5c48f347fe8a8309e4810a6015: Sending DeleteTablet for 3 replicas of tablet 0e92744ce8794d609be0b33a9240e372
I20260928 06:37:19.038555 2790 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e92744ce8794d609be0b33a9240e372 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:37:19 UTC) from {username='kudu', principal='kudu/127.16.157.126@KRBTEST.COM'} at 127.0.0.1:43926
I20260928 06:37:19.038578 2924 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e92744ce8794d609be0b33a9240e372 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:37:19 UTC) from {username='kudu', principal='kudu/127.16.157.126@KRBTEST.COM'} at 127.0.0.1:48474
I20260928 06:37:19.038636 3058 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e92744ce8794d609be0b33a9240e372 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:37:19 UTC) from {username='kudu', principal='kudu/127.16.157.126@KRBTEST.COM'} at 127.0.0.1:47178
I20260928 06:37:19.038803 3161 tablet_replica.cc:330] T 0e92744ce8794d609be0b33a9240e372 P 59cb73ff0f3a40f78a70ba7d246eee0d: stopping tablet replica
I20260928 06:37:19.038889 3161 raft_consensus.cc:2229] T 0e92744ce8794d609be0b33a9240e372 P 59cb73ff0f3a40f78a70ba7d246eee0d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:37:19.038904 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 2727
I20260928 06:37:19.038987 3161 raft_consensus.cc:2258] T 0e92744ce8794d609be0b33a9240e372 P 59cb73ff0f3a40f78a70ba7d246eee0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:37:19.044231 3160 tablet_replica.cc:330] T 0e92744ce8794d609be0b33a9240e372 P d71741fa823b4a28819d28d5af7af200: stopping tablet replica
I20260928 06:37:19.044337 3160 raft_consensus.cc:2229] T 0e92744ce8794d609be0b33a9240e372 P d71741fa823b4a28819d28d5af7af200 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:37:19.044431 3160 raft_consensus.cc:2258] T 0e92744ce8794d609be0b33a9240e372 P d71741fa823b4a28819d28d5af7af200 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:37:19.044588 3162 tablet_replica.cc:330] T 0e92744ce8794d609be0b33a9240e372 P d0f3323eaa4b42f083730f7c2f294786: stopping tablet replica
I20260928 06:37:19.044685 3162 raft_consensus.cc:2229] T 0e92744ce8794d609be0b33a9240e372 P d0f3323eaa4b42f083730f7c2f294786 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:37:19.044767 3162 raft_consensus.cc:2258] T 0e92744ce8794d609be0b33a9240e372 P d0f3323eaa4b42f083730f7c2f294786 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:37:19.045171 3160 ts_tablet_manager.cc:1905] T 0e92744ce8794d609be0b33a9240e372 P d71741fa823b4a28819d28d5af7af200: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:37:19.045169 3162 ts_tablet_manager.cc:1905] T 0e92744ce8794d609be0b33a9240e372 P d0f3323eaa4b42f083730f7c2f294786: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 06:37:19.046303 2649 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.157.65:43235 (error 108)
I20260928 06:37:19.047039 3133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:19.046910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.65:43235 (local address 127.0.0.1:43942)
0928 06:37:19.046959 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:19.047006 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.65:43235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 06:37:19.047134 3162 ts_tablet_manager.cc:1918] T 0e92744ce8794d609be0b33a9240e372 P d0f3323eaa4b42f083730f7c2f294786: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:37:19.047197 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 2861
I20260928 06:37:19.047224 3162 log.cc:1196] T 0e92744ce8794d609be0b33a9240e372 P d0f3323eaa4b42f083730f7c2f294786: Deleting WAL directory at /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577196946045-17013-0/minicluster-data/ts-2/wal/wals/0e92744ce8794d609be0b33a9240e372
W20260928 06:37:19.047274 2649 catalog_manager.cc:4584] TS 59cb73ff0f3a40f78a70ba7d246eee0d (127.16.157.65:43235): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0e92744ce8794d609be0b33a9240e372: Network error: Client connection negotiation failed: client connection to 127.16.157.65:43235: connect: Connection refused (error 111)
I20260928 06:37:19.047483 3162 ts_tablet_manager.cc:1939] T 0e92744ce8794d609be0b33a9240e372 P d0f3323eaa4b42f083730f7c2f294786: Deleting consensus metadata
I20260928 06:37:19.048856 2652 catalog_manager.cc:4841] TS d0f3323eaa4b42f083730f7c2f294786 (127.16.157.67:42127): tablet 0e92744ce8794d609be0b33a9240e372 (table test-table [id=bd04737a223c446186c11a565b892ab3]) successfully deleted
I20260928 06:37:19.052664 3133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:19.052517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.66:40297 (local address 127.0.0.1:48482)
0928 06:37:19.052596 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:19.052626 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.66:40297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 06:37:19.052821 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 2995
I20260928 06:37:19.058538 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 2633
2026-09-28T06:37:19Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1934 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T06:37:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:37:19Z Disabled control of system clock
I20260928 06:37:19.086782 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577196946045-17013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577196946045-17013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577196946045-17013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:39425
--webserver_interface=127.16.157.126
--webserver_port=0
--builtin_ntp_servers=127.16.157.84:45141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.157.126:39425
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 06:37:19.166074 3167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:19.166404 3167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:19.166432 3167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:19.167636 3167 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:37:19.167682 3167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:19.167698 3167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:19.167752 3167 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:37:19.167773 3167 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:37:19.169150 3167 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:45141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577196946045-17013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577196946045-17013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.157.126:39425
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:39425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577196946045-17013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:37:19.169409 3167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:19.169754 3167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:19.174705 3172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:19.174683 3173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:19.174932 3167 server_base.cc:1034] running on GCE node
W20260928 06:37:19.174683 3175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:19.175248 3167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:19.175580 3167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:19.176749 3167 hybrid_clock.cc:648] HybridClock initialized: now 1790577439176725 us; error 35 us; skew 500 ppm
I20260928 06:37:19.184208 3167 webserver.cc:460] Webserver started at http://127.16.157.126:43871/ using document root <none> and password file <none>
I20260928 06:37:19.184491 3167 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:19.184567 3167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:19.184720 3167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:37:19.185871 3167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577196946045-17013-0/minicluster-data/master-0/data/instance:
uuid: "c8fd5e2a55e84a12a649de0b000b8884"
format_stamp: "Formatted at 2026-09-28 06:37:19 on dist-test-slave-rrj5"
server_key: "1c35c33e23650c472b462911642f2c49"
server_key_iv: "11c49e9c170eaf77681929703e39e10b"
server_key_version: "encryptionkey@0"
I20260928 06:37:19.186228 3167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577196946045-17013-0/minicluster-data/master-0/wal/instance:
uuid: "c8fd5e2a55e84a12a649de0b000b8884"
format_stamp: "Formatted at 2026-09-28 06:37:19 on dist-test-slave-rrj5"
server_key: "1c35c33e23650c472b462911642f2c49"
server_key_iv: "11c49e9c170eaf77681929703e39e10b"
server_key_version: "encryptionkey@0"
I20260928 06:37:19.187522 3167 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:37:19.188230 3181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:19.188407 3167 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:37:19.188516 3167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577196946045-17013-0/minicluster-data/master-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577196946045-17013-0/minicluster-data/master-0/wal
uuid: "c8fd5e2a55e84a12a649de0b000b8884"
format_stamp: "Formatted at 2026-09-28 06:37:19 on dist-test-slave-rrj5"
server_key: "1c35c33e23650c472b462911642f2c49"
server_key_iv: "11c49e9c170eaf77681929703e39e10b"
server_key_version: "encryptionkey@0"
I20260928 06:37:19.188592 3167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577196946045-17013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577196946045-17013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577196946045-17013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:19.198901 3167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:19.199189 3167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:19.199334 3167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:19.203222 3167 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.126:39425
I20260928 06:37:19.203258 3233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.126:39425 every 8 connection(s)
I20260928 06:37:19.203562 3167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
I20260928 06:37:19.204463 3234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:37:19.206718 3234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8fd5e2a55e84a12a649de0b000b8884: Bootstrap starting.
I20260928 06:37:19.207271 3234 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8fd5e2a55e84a12a649de0b000b8884: Neither blocks nor log segments found. Creating new log.
I20260928 06:37:19.207568 3234 log.cc:824] T 00000000000000000000000000000000 P c8fd5e2a55e84a12a649de0b000b8884: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:19.208201 3234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8fd5e2a55e84a12a649de0b000b8884: No bootstrap required, opened a new log
I20260928 06:37:19.209797 3234 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8fd5e2a55e84a12a649de0b000b8884 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8fd5e2a55e84a12a649de0b000b8884" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 39425 } }
I20260928 06:37:19.209924 3234 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8fd5e2a55e84a12a649de0b000b8884 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:37:19.209945 3234 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8fd5e2a55e84a12a649de0b000b8884 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8fd5e2a55e84a12a649de0b000b8884, State: Initialized, Role: FOLLOWER
I20260928 06:37:19.210119 3234 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8fd5e2a55e84a12a649de0b000b8884 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8fd5e2a55e84a12a649de0b000b8884" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 39425 } }
I20260928 06:37:19.210202 3234 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8fd5e2a55e84a12a649de0b000b8884 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:37:19.210247 3234 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8fd5e2a55e84a12a649de0b000b8884 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:37:19.210301 3234 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8fd5e2a55e84a12a649de0b000b8884 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:37:19.210805 3234 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8fd5e2a55e84a12a649de0b000b8884 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8fd5e2a55e84a12a649de0b000b8884" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 39425 } }
I20260928 06:37:19.210883 3234 leader_election.cc:302] T 00000000000000000000000000000000 P c8fd5e2a55e84a12a649de0b000b8884 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8fd5e2a55e84a12a649de0b000b8884; no voters:
I20260928 06:37:19.210975 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 3167
I20260928 06:37:19.211079 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577196946045-17013-0/minicluster-data/master-0/wal/instance
I20260928 06:37:19.211089 3234 leader_election.cc:288] T 00000000000000000000000000000000 P c8fd5e2a55e84a12a649de0b000b8884 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:37:19.211118 3237 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8fd5e2a55e84a12a649de0b000b8884 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:37:19.211259 3237 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8fd5e2a55e84a12a649de0b000b8884 [term 1 LEADER]: Becoming Leader. State: Replica: c8fd5e2a55e84a12a649de0b000b8884, State: Running, Role: LEADER
I20260928 06:37:19.211314 17013 external_mini_cluster.cc:1428] Setting key 361fe914094f266d016c033b4e050663
I20260928 06:37:19.211341 3234 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8fd5e2a55e84a12a649de0b000b8884 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:37:19.211349 3237 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8fd5e2a55e84a12a649de0b000b8884 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8fd5e2a55e84a12a649de0b000b8884" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 39425 } }
I20260928 06:37:19.212473 3238 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8fd5e2a55e84a12a649de0b000b8884 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8fd5e2a55e84a12a649de0b000b8884. Latest consensus state: current_term: 1 leader_uuid: "c8fd5e2a55e84a12a649de0b000b8884" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8fd5e2a55e84a12a649de0b000b8884" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 39425 } } }
I20260928 06:37:19.212570 3238 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8fd5e2a55e84a12a649de0b000b8884 [sys.catalog]: This master's current role is: LEADER
I20260928 06:37:19.212882 3239 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8fd5e2a55e84a12a649de0b000b8884 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8fd5e2a55e84a12a649de0b000b8884" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8fd5e2a55e84a12a649de0b000b8884" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 39425 } } }
I20260928 06:37:19.213021 3239 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8fd5e2a55e84a12a649de0b000b8884 [sys.catalog]: This master's current role is: LEADER
W20260928 06:37:19.213275 3254 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c8fd5e2a55e84a12a649de0b000b8884: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:37:19.213335 3254 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:37:19.213411 3252 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:37:19.213687 3252 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:37:19.215330 3252 catalog_manager.cc:1347] Generated new cluster ID: 15b74fcf597c47d19e3c71c569f70bc3
I20260928 06:37:19.215379 3252 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:37:19.218101 3242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:19.211753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51220 (local address 127.16.157.126:39425)
0928 06:37:19.212078 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:19.212087 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:19.212116 (+ 29us) server_negotiation.cc:407] Connection header received
0928 06:37:19.212228 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:19.212254 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:19.212347 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:19.213154 (+ 807us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:19.213252 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:19.214651 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:19.215765 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:19.216443 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:19.216572 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:19.217336 (+ 764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:19.217340 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:19.217341 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:37:19.217567 (+ 226us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:19.217638 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:37:19.217653 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:19.217723 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:19.217797 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 06:37:19.217880 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":95,"threads_started":1}
I20260928 06:37:19.228116 3252 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:37:19.228817 3252 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:37:19.240371 3252 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8fd5e2a55e84a12a649de0b000b8884: Generated new TSK 0
I20260928 06:37:19.240620 3252 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:37:19.246788 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577196946045-17013-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577196946045-17013-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577196946045-17013-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.65:0
--local_ip_for_outbound_sockets=127.16.157.65
--webserver_interface=127.16.157.65
--webserver_port=0
--tserver_master_addrs=127.16.157.126:39425
--builtin_ntp_servers=127.16.157.84:45141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 06:37:19.331022 3258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:19.331292 3258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:19.331331 3258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:19.332507 3258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:19.332587 3258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:19.332631 3258 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.65
I20260928 06:37:19.334107 3258 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:45141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577196946045-17013-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577196946045-17013-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.157.65
--webserver_port=0
--tserver_master_addrs=127.16.157.126:39425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.65
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577196946045-17013-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:37:19.334380 3258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:19.334694 3258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:19.339890 3263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:19.340065 3258 server_base.cc:1034] running on GCE node
W20260928 06:37:19.339910 3266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:19.339895 3264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:19.340446 3258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:19.340767 3258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:19.341919 3258 hybrid_clock.cc:648] HybridClock initialized: now 1790577439341894 us; error 41 us; skew 500 ppm
I20260928 06:37:19.343281 3258 webserver.cc:460] Webserver started at http://127.16.157.65:37793/ using document root <none> and password file <none>
I20260928 06:37:19.343544 3258 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:19.343616 3258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:19.343798 3258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:37:19.344897 3258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577196946045-17013-0/minicluster-data/ts-0/data/instance:
uuid: "215645205d7e428f821c22ca54aab1ee"
format_stamp: "Formatted at 2026-09-28 06:37:19 on dist-test-slave-rrj5"
server_key: "6bb6fd470fa5ebdbf36301274ab112ec"
server_key_iv: "388c2984a8373064307c3d904c971da1"
server_key_version: "encryptionkey@0"
I20260928 06:37:19.345281 3258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577196946045-17013-0/minicluster-data/ts-0/wal/instance:
uuid: "215645205d7e428f821c22ca54aab1ee"
format_stamp: "Formatted at 2026-09-28 06:37:19 on dist-test-slave-rrj5"
server_key: "6bb6fd470fa5ebdbf36301274ab112ec"
server_key_iv: "388c2984a8373064307c3d904c971da1"
server_key_version: "encryptionkey@0"
I20260928 06:37:19.346707 3258 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:37:19.347476 3272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:19.347661 3258 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:37:19.347782 3258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577196946045-17013-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577196946045-17013-0/minicluster-data/ts-0/wal
uuid: "215645205d7e428f821c22ca54aab1ee"
format_stamp: "Formatted at 2026-09-28 06:37:19 on dist-test-slave-rrj5"
server_key: "6bb6fd470fa5ebdbf36301274ab112ec"
server_key_iv: "388c2984a8373064307c3d904c971da1"
server_key_version: "encryptionkey@0"
I20260928 06:37:19.347857 3258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577196946045-17013-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577196946045-17013-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577196946045-17013-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:19.358987 3258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:19.359308 3258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:19.359464 3258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:19.359702 3258 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:37:19.360026 3258 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:37:19.360081 3258 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:19.360124 3258 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:37:19.360149 3258 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:19.366516 3258 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.65:40461
I20260928 06:37:19.366614 3385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.65:40461 every 8 connection(s)
I20260928 06:37:19.366946 3258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
I20260928 06:37:19.372426 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 3258
I20260928 06:37:19.372555 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577196946045-17013-0/minicluster-data/ts-0/wal/instance
I20260928 06:37:19.372582 3242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:19.367745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:54535 (local address 127.16.157.126:39425)
0928 06:37:19.367817 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:19.367820 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:19.368083 (+ 263us) server_negotiation.cc:407] Connection header received
0928 06:37:19.368619 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:19.368620 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:19.368632 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:19.368684 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:19.369384 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:19.370184 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:19.371184 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:19.371382 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:19.371442 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:19.372253 (+ 811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:19.372255 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:19.372255 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:37:19.372261 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:19.372282 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:37:19.372288 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:19.372361 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:19.372442 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 06:37:19.372480 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 06:37:19.372778 17013 external_mini_cluster.cc:1428] Setting key 419cd76d258fc1f1d9492b0d609b38c6
I20260928 06:37:19.372786 3388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:19.367742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:39425 (local address 127.16.157.65:54535)
0928 06:37:19.367961 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:19.367995 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:19.368514 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:19.368746 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:19.368749 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:19.368758 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:37:19.369296 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:19.369301 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:19.370279 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:19.370280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:19.371087 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:19.371089 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:19.371462 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:19.371462 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:19.371573 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:19.372089 (+ 516us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:37:19.372089 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:37:19.372120 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:37:19.372143 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:37:19.372360 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:37:19.372365 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:37:19.372368 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:37:19.372421 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 06:37:19.372533 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":92,"threads_started":1}
I20260928 06:37:19.373368 3386 heartbeater.cc:347] Connected to a master server at 127.16.157.126:39425
I20260928 06:37:19.373479 3386 heartbeater.cc:464] Registering TS with master...
I20260928 06:37:19.373721 3386 heartbeater.cc:511] Master 127.16.157.126:39425 requested a full tablet report, sending...
I20260928 06:37:19.373998 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577196946045-17013-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577196946045-17013-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577196946045-17013-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.66:0
--local_ip_for_outbound_sockets=127.16.157.66
--webserver_interface=127.16.157.66
--webserver_port=0
--tserver_master_addrs=127.16.157.126:39425
--builtin_ntp_servers=127.16.157.84:45141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 06:37:19.374302 3197 ts_manager.cc:194] Registered new tserver with Master: 215645205d7e428f821c22ca54aab1ee (127.16.157.65:40461)
I20260928 06:37:19.375231 3197 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.157.65:54535
W20260928 06:37:19.459769 3389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:19.459957 3389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:19.459978 3389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:19.461143 3389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:19.461184 3389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:19.461208 3389 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.66
I20260928 06:37:19.462656 3389 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:45141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577196946045-17013-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577196946045-17013-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.157.66
--webserver_port=0
--tserver_master_addrs=127.16.157.126:39425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.66
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577196946045-17013-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:37:19.462930 3389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:19.463243 3389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:19.468439 3394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:19.468415 3397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:19.468431 3395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:19.468736 3389 server_base.cc:1034] running on GCE node
I20260928 06:37:19.468957 3389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:19.469223 3389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:19.470382 3389 hybrid_clock.cc:648] HybridClock initialized: now 1790577439470362 us; error 30 us; skew 500 ppm
I20260928 06:37:19.471666 3389 webserver.cc:460] Webserver started at http://127.16.157.66:33345/ using document root <none> and password file <none>
I20260928 06:37:19.471930 3389 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:19.471978 3389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:19.472150 3389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:37:19.473250 3389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577196946045-17013-0/minicluster-data/ts-1/data/instance:
uuid: "fcb1729bd2424930bc5bebc205724683"
format_stamp: "Formatted at 2026-09-28 06:37:19 on dist-test-slave-rrj5"
server_key: "7562fac11c5ae3bffb030177ab159afa"
server_key_iv: "0be5a803e0a6b69e7bae7d280d0f9cb3"
server_key_version: "encryptionkey@0"
I20260928 06:37:19.473655 3389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577196946045-17013-0/minicluster-data/ts-1/wal/instance:
uuid: "fcb1729bd2424930bc5bebc205724683"
format_stamp: "Formatted at 2026-09-28 06:37:19 on dist-test-slave-rrj5"
server_key: "7562fac11c5ae3bffb030177ab159afa"
server_key_iv: "0be5a803e0a6b69e7bae7d280d0f9cb3"
server_key_version: "encryptionkey@0"
I20260928 06:37:19.474982 3389 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:37:19.475754 3403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:19.475931 3389 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:37:19.476049 3389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577196946045-17013-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577196946045-17013-0/minicluster-data/ts-1/wal
uuid: "fcb1729bd2424930bc5bebc205724683"
format_stamp: "Formatted at 2026-09-28 06:37:19 on dist-test-slave-rrj5"
server_key: "7562fac11c5ae3bffb030177ab159afa"
server_key_iv: "0be5a803e0a6b69e7bae7d280d0f9cb3"
server_key_version: "encryptionkey@0"
I20260928 06:37:19.476146 3389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577196946045-17013-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577196946045-17013-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577196946045-17013-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:19.488516 3389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:19.488830 3389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:19.488981 3389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:19.489216 3389 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:37:19.489570 3389 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:37:19.489650 3389 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:19.489682 3389 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:37:19.489710 3389 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:19.495805 3389 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.66:39087
I20260928 06:37:19.495894 3516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.66:39087 every 8 connection(s)
I20260928 06:37:19.496232 3389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
I20260928 06:37:19.499873 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 3389
I20260928 06:37:19.499992 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577196946045-17013-0/minicluster-data/ts-1/wal/instance
I20260928 06:37:19.500221 17013 external_mini_cluster.cc:1428] Setting key 5f48d0eb3670c995d1292b5d813fb0d0
I20260928 06:37:19.501235 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577196946045-17013-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577196946045-17013-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577196946045-17013-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.67:0
--local_ip_for_outbound_sockets=127.16.157.67
--webserver_interface=127.16.157.67
--webserver_port=0
--tserver_master_addrs=127.16.157.126:39425
--builtin_ntp_servers=127.16.157.84:45141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 06:37:19.501920 3242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:19.497006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.66:52601 (local address 127.16.157.126:39425)
0928 06:37:19.497081 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:19.497084 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:19.497342 (+ 258us) server_negotiation.cc:407] Connection header received
0928 06:37:19.497923 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:19.497924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:19.497934 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:19.497991 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:19.498646 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:19.499138 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:19.500190 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:19.500493 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:19.500569 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:19.501639 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:19.501640 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:19.501641 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:37:19.501645 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:19.501667 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:37:19.501671 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:19.501719 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:19.501839 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 06:37:19.501863 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 06:37:19.502218 3519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:19.496982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:39425 (local address 127.16.157.66:52601)
0928 06:37:19.497216 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:19.497251 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:19.497831 (+ 580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:19.498070 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:19.498073 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:19.498081 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:37:19.498573 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:19.498577 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:19.499219 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:19.499220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:19.500093 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:19.500096 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:19.500597 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:19.500599 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:19.500766 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:19.501426 (+ 660us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:37:19.501427 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:37:19.501466 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:37:19.501494 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:37:19.501741 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:37:19.501748 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:37:19.501751 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:37:19.501834 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 06:37:19.501951 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":125,"threads_started":1}
I20260928 06:37:19.502813 3517 heartbeater.cc:347] Connected to a master server at 127.16.157.126:39425
I20260928 06:37:19.502928 3517 heartbeater.cc:464] Registering TS with master...
I20260928 06:37:19.503134 3517 heartbeater.cc:511] Master 127.16.157.126:39425 requested a full tablet report, sending...
I20260928 06:37:19.503537 3197 ts_manager.cc:194] Registered new tserver with Master: fcb1729bd2424930bc5bebc205724683 (127.16.157.66:39087)
I20260928 06:37:19.504067 3197 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.157.66:52601
W20260928 06:37:19.585155 3520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:19.585415 3520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:19.585456 3520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:19.586745 3520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:19.586814 3520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:19.586859 3520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.67
I20260928 06:37:19.588239 3520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:45141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577196946045-17013-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577196946045-17013-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.157.67
--webserver_port=0
--tserver_master_addrs=127.16.157.126:39425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.67
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577196946045-17013-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:37:19.588515 3520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:19.588825 3520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:19.593935 3525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:19.593982 3526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:19.593957 3528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:19.594187 3520 server_base.cc:1034] running on GCE node
I20260928 06:37:19.594401 3520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:19.594691 3520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:19.595857 3520 hybrid_clock.cc:648] HybridClock initialized: now 1790577439595838 us; error 36 us; skew 500 ppm
I20260928 06:37:19.597190 3520 webserver.cc:460] Webserver started at http://127.16.157.67:46211/ using document root <none> and password file <none>
I20260928 06:37:19.597447 3520 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:19.597518 3520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:19.597699 3520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:37:19.598773 3520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577196946045-17013-0/minicluster-data/ts-2/data/instance:
uuid: "18513438818e4a07954992bc9a60c4e8"
format_stamp: "Formatted at 2026-09-28 06:37:19 on dist-test-slave-rrj5"
server_key: "ad563a6f8398374d9a71d16eaeec07c1"
server_key_iv: "01a350b8df98ff2c1252506ac6e0765e"
server_key_version: "encryptionkey@0"
I20260928 06:37:19.599148 3520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577196946045-17013-0/minicluster-data/ts-2/wal/instance:
uuid: "18513438818e4a07954992bc9a60c4e8"
format_stamp: "Formatted at 2026-09-28 06:37:19 on dist-test-slave-rrj5"
server_key: "ad563a6f8398374d9a71d16eaeec07c1"
server_key_iv: "01a350b8df98ff2c1252506ac6e0765e"
server_key_version: "encryptionkey@0"
I20260928 06:37:19.600430 3520 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:37:19.601138 3534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:19.601321 3520 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:37:19.601418 3520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577196946045-17013-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577196946045-17013-0/minicluster-data/ts-2/wal
uuid: "18513438818e4a07954992bc9a60c4e8"
format_stamp: "Formatted at 2026-09-28 06:37:19 on dist-test-slave-rrj5"
server_key: "ad563a6f8398374d9a71d16eaeec07c1"
server_key_iv: "01a350b8df98ff2c1252506ac6e0765e"
server_key_version: "encryptionkey@0"
I20260928 06:37:19.601500 3520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577196946045-17013-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577196946045-17013-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577196946045-17013-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:19.609972 3520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:19.610235 3520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:19.610378 3520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:19.610581 3520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:37:19.610891 3520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:37:19.610947 3520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:19.611013 3520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:37:19.611047 3520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:19.616897 3520 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.67:33619
I20260928 06:37:19.616956 3647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.67:33619 every 8 connection(s)
I20260928 06:37:19.617257 3520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
I20260928 06:37:19.622280 3242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:19.618032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:49285 (local address 127.16.157.126:39425)
0928 06:37:19.618098 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:19.618100 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:19.618353 (+ 253us) server_negotiation.cc:407] Connection header received
0928 06:37:19.618859 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:19.618861 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:19.618870 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:19.618930 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:19.619573 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:19.620058 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:19.620991 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:19.621180 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:19.621229 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:19.622039 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:19.622040 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:19.622040 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:37:19.622044 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:19.622058 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:37:19.622061 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:19.622104 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:19.622195 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 06:37:19.622222 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 06:37:19.622455 3650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:19.618008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:39425 (local address 127.16.157.67:49285)
0928 06:37:19.618213 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:19.618253 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:19.618747 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:19.618981 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:19.618984 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:19.618991 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:37:19.619498 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:19.619501 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:19.620148 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:19.620149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:19.620901 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:19.620903 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:19.621254 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:19.621255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:19.621370 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:19.621908 (+ 538us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:37:19.621908 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:37:19.621936 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:37:19.621958 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:37:19.622116 (+ 158us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:37:19.622121 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:37:19.622123 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:37:19.622174 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 06:37:19.622252 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":108,"threads_started":1}
I20260928 06:37:19.622934 3648 heartbeater.cc:347] Connected to a master server at 127.16.157.126:39425
I20260928 06:37:19.623036 3648 heartbeater.cc:464] Registering TS with master...
I20260928 06:37:19.623209 3648 heartbeater.cc:511] Master 127.16.157.126:39425 requested a full tablet report, sending...
I20260928 06:37:19.623595 3197 ts_manager.cc:194] Registered new tserver with Master: 18513438818e4a07954992bc9a60c4e8 (127.16.157.67:33619)
I20260928 06:37:19.623948 3197 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.157.67:49285
I20260928 06:37:19.627168 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 3520
I20260928 06:37:19.627291 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577196946045-17013-0/minicluster-data/ts-2/wal/instance
I20260928 06:37:19.627516 17013 external_mini_cluster.cc:1428] Setting key 877c1045a9b21d67b05bfb4484c62deb
I20260928 06:37:19.628459 17013 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:37:19.632468 3242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:19.629852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51234 (local address 127.16.157.126:39425)
0928 06:37:19.629911 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:19.629912 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:19.630015 (+ 103us) server_negotiation.cc:407] Connection header received
0928 06:37:19.630060 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:19.630061 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:19.630072 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:19.630126 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:19.630270 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:19.630708 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:19.631396 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:19.631568 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:19.631616 (+ 48us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:19.632286 (+ 670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:19.632286 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:19.632287 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:37:19.632290 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:19.632301 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:37:19.632304 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:19.632366 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:19.632409 (+ 43us) server_negotiation.cc:299] Negotiation successful
0928 06:37:19.632418 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 06:37:19.633960 3197 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51234:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:37:19.635412 3197 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:37:19.641510 3658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:19.637430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.67:33619 (local address 127.0.0.1:58584)
0928 06:37:19.637725 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:19.637772 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:19.637850 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:19.638072 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:19.638075 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:19.638077 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:37:19.638132 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:19.638133 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:19.639182 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:19.639182 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:19.640191 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:19.640194 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:19.641212 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:19.641213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:19.641304 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:19.641317 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 06:37:19.641384 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 06:37:19.641432 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":71,"threads_started":1}
I20260928 06:37:19.641688 3659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:19.637490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58584 (local address 127.16.157.67:33619)
0928 06:37:19.637648 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:19.637653 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:19.637860 (+ 207us) server_negotiation.cc:407] Connection header received
0928 06:37:19.637948 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:19.637952 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:19.637968 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:19.638056 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:37:19.638288 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:19.639101 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:19.640297 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:19.641021 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:19.641066 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:19.641149 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:19.641420 (+ 271us) server_negotiation.cc:299] Negotiation successful
0928 06:37:19.641461 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":97,"threads_started":1}
I20260928 06:37:19.641965 3656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:19.637433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.66:39087 (local address 127.0.0.1:37210)
0928 06:37:19.637725 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:19.637764 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:19.637850 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:19.638376 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:19.638377 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:19.638379 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:37:19.638422 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:19.638423 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:19.639702 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:19.639703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:19.640693 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:19.640695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:19.641548 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:19.641549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:19.641785 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:19.641790 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:37:19.641852 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 06:37:19.641892 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":11,"thread_start_us":84,"threads_started":1}
I20260928 06:37:19.642225 3660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:19.637712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37210 (local address 127.16.157.66:39087)
0928 06:37:19.638160 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:19.638164 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:19.638185 (+ 21us) server_negotiation.cc:407] Connection header received
0928 06:37:19.638222 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:19.638224 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:19.638238 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:19.638311 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:37:19.638499 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:19.639617 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:19.640771 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:19.641332 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:19.641421 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:19.641509 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:19.642073 (+ 564us) server_negotiation.cc:299] Negotiation successful
0928 06:37:19.642118 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":120,"threads_started":1}
I20260928 06:37:19.642378 3582 tablet_service.cc:1467] Processing CreateTablet for tablet 83caee4530ca4cad846420cc4b7157e0 (DEFAULT_TABLE table=test-table [id=b9c96980124441b6832710b0c3f2189c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:37:19.642911 3582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83caee4530ca4cad846420cc4b7157e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:37:19.643209 3451 tablet_service.cc:1467] Processing CreateTablet for tablet 83caee4530ca4cad846420cc4b7157e0 (DEFAULT_TABLE table=test-table [id=b9c96980124441b6832710b0c3f2189c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:37:19.643358 3657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:19.637437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.65:40461 (local address 127.0.0.1:54014)
0928 06:37:19.637951 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:19.637958 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:19.637999 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:19.638376 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:19.638377 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:19.638381 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:37:19.638419 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:19.638421 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:19.640792 (+ 2371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:19.640793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:19.642060 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:19.642064 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:19.643096 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:19.643097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:19.643154 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:19.643158 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:37:19.643207 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 06:37:19.643282 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"mutex_wait_us":144,"thread_start_us":90,"threads_started":1}
I20260928 06:37:19.643473 3451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83caee4530ca4cad846420cc4b7157e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:37:19.643663 3661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:19.637939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54014 (local address 127.16.157.65:40461)
0928 06:37:19.638152 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:19.638157 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:19.638167 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:37:19.638216 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:19.638218 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:19.638233 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:19.638311 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:37:19.638500 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:19.639606 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:19.642159 (+ 2553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:19.642902 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:19.642970 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:19.643059 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:19.643531 (+ 472us) server_negotiation.cc:299] Negotiation successful
0928 06:37:19.643580 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":91,"threads_started":1}
I20260928 06:37:19.644377 3320 tablet_service.cc:1467] Processing CreateTablet for tablet 83caee4530ca4cad846420cc4b7157e0 (DEFAULT_TABLE table=test-table [id=b9c96980124441b6832710b0c3f2189c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:37:19.644686 3320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83caee4530ca4cad846420cc4b7157e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:37:19.646148 3662 tablet_bootstrap.cc:492] T 83caee4530ca4cad846420cc4b7157e0 P fcb1729bd2424930bc5bebc205724683: Bootstrap starting.
I20260928 06:37:19.646795 3663 tablet_bootstrap.cc:492] T 83caee4530ca4cad846420cc4b7157e0 P 18513438818e4a07954992bc9a60c4e8: Bootstrap starting.
I20260928 06:37:19.646826 3662 tablet_bootstrap.cc:654] T 83caee4530ca4cad846420cc4b7157e0 P fcb1729bd2424930bc5bebc205724683: Neither blocks nor log segments found. Creating new log.
I20260928 06:37:19.647128 3662 log.cc:824] T 83caee4530ca4cad846420cc4b7157e0 P fcb1729bd2424930bc5bebc205724683: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:19.647428 3663 tablet_bootstrap.cc:654] T 83caee4530ca4cad846420cc4b7157e0 P 18513438818e4a07954992bc9a60c4e8: Neither blocks nor log segments found. Creating new log.
I20260928 06:37:19.647787 3663 log.cc:824] T 83caee4530ca4cad846420cc4b7157e0 P 18513438818e4a07954992bc9a60c4e8: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:19.647843 3662 tablet_bootstrap.cc:492] T 83caee4530ca4cad846420cc4b7157e0 P fcb1729bd2424930bc5bebc205724683: No bootstrap required, opened a new log
I20260928 06:37:19.647922 3662 ts_tablet_manager.cc:1397] T 83caee4530ca4cad846420cc4b7157e0 P fcb1729bd2424930bc5bebc205724683: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:37:19.648149 3666 tablet_bootstrap.cc:492] T 83caee4530ca4cad846420cc4b7157e0 P 215645205d7e428f821c22ca54aab1ee: Bootstrap starting.
I20260928 06:37:19.648576 3663 tablet_bootstrap.cc:492] T 83caee4530ca4cad846420cc4b7157e0 P 18513438818e4a07954992bc9a60c4e8: No bootstrap required, opened a new log
I20260928 06:37:19.648644 3663 ts_tablet_manager.cc:1397] T 83caee4530ca4cad846420cc4b7157e0 P 18513438818e4a07954992bc9a60c4e8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:37:19.648687 3666 tablet_bootstrap.cc:654] T 83caee4530ca4cad846420cc4b7157e0 P 215645205d7e428f821c22ca54aab1ee: Neither blocks nor log segments found. Creating new log.
I20260928 06:37:19.648942 3666 log.cc:824] T 83caee4530ca4cad846420cc4b7157e0 P 215645205d7e428f821c22ca54aab1ee: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:19.649652 3666 tablet_bootstrap.cc:492] T 83caee4530ca4cad846420cc4b7157e0 P 215645205d7e428f821c22ca54aab1ee: No bootstrap required, opened a new log
I20260928 06:37:19.649722 3666 ts_tablet_manager.cc:1397] T 83caee4530ca4cad846420cc4b7157e0 P 215645205d7e428f821c22ca54aab1ee: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:37:19.650755 3662 raft_consensus.cc:348] T 83caee4530ca4cad846420cc4b7157e0 P fcb1729bd2424930bc5bebc205724683 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb1729bd2424930bc5bebc205724683" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 39087 } } peers { permanent_uuid: "18513438818e4a07954992bc9a60c4e8" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 33619 } } peers { permanent_uuid: "215645205d7e428f821c22ca54aab1ee" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 40461 } }
I20260928 06:37:19.650954 3662 raft_consensus.cc:374] T 83caee4530ca4cad846420cc4b7157e0 P fcb1729bd2424930bc5bebc205724683 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:37:19.650997 3662 raft_consensus.cc:729] T 83caee4530ca4cad846420cc4b7157e0 P fcb1729bd2424930bc5bebc205724683 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcb1729bd2424930bc5bebc205724683, State: Initialized, Role: FOLLOWER
I20260928 06:37:19.651113 3662 consensus_queue.cc:260] T 83caee4530ca4cad846420cc4b7157e0 P fcb1729bd2424930bc5bebc205724683 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb1729bd2424930bc5bebc205724683" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 39087 } } peers { permanent_uuid: "18513438818e4a07954992bc9a60c4e8" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 33619 } } peers { permanent_uuid: "215645205d7e428f821c22ca54aab1ee" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 40461 } }
I20260928 06:37:19.651377 3662 ts_tablet_manager.cc:1428] T 83caee4530ca4cad846420cc4b7157e0 P fcb1729bd2424930bc5bebc205724683: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 06:37:19.651399 3663 raft_consensus.cc:348] T 83caee4530ca4cad846420cc4b7157e0 P 18513438818e4a07954992bc9a60c4e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb1729bd2424930bc5bebc205724683" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 39087 } } peers { permanent_uuid: "18513438818e4a07954992bc9a60c4e8" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 33619 } } peers { permanent_uuid: "215645205d7e428f821c22ca54aab1ee" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 40461 } }
I20260928 06:37:19.651535 3663 raft_consensus.cc:374] T 83caee4530ca4cad846420cc4b7157e0 P 18513438818e4a07954992bc9a60c4e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:37:19.651574 3663 raft_consensus.cc:729] T 83caee4530ca4cad846420cc4b7157e0 P 18513438818e4a07954992bc9a60c4e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18513438818e4a07954992bc9a60c4e8, State: Initialized, Role: FOLLOWER
I20260928 06:37:19.651583 3517 heartbeater.cc:503] Master 127.16.157.126:39425 was elected leader, sending a full tablet report...
I20260928 06:37:19.651686 3663 consensus_queue.cc:260] T 83caee4530ca4cad846420cc4b7157e0 P 18513438818e4a07954992bc9a60c4e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb1729bd2424930bc5bebc205724683" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 39087 } } peers { permanent_uuid: "18513438818e4a07954992bc9a60c4e8" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 33619 } } peers { permanent_uuid: "215645205d7e428f821c22ca54aab1ee" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 40461 } }
I20260928 06:37:19.651929 3663 ts_tablet_manager.cc:1428] T 83caee4530ca4cad846420cc4b7157e0 P 18513438818e4a07954992bc9a60c4e8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:37:19.652058 3666 raft_consensus.cc:348] T 83caee4530ca4cad846420cc4b7157e0 P 215645205d7e428f821c22ca54aab1ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb1729bd2424930bc5bebc205724683" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 39087 } } peers { permanent_uuid: "18513438818e4a07954992bc9a60c4e8" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 33619 } } peers { permanent_uuid: "215645205d7e428f821c22ca54aab1ee" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 40461 } }
I20260928 06:37:19.652195 3666 raft_consensus.cc:374] T 83caee4530ca4cad846420cc4b7157e0 P 215645205d7e428f821c22ca54aab1ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:37:19.652227 3666 raft_consensus.cc:729] T 83caee4530ca4cad846420cc4b7157e0 P 215645205d7e428f821c22ca54aab1ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 215645205d7e428f821c22ca54aab1ee, State: Initialized, Role: FOLLOWER
I20260928 06:37:19.652333 3648 heartbeater.cc:503] Master 127.16.157.126:39425 was elected leader, sending a full tablet report...
I20260928 06:37:19.652328 3666 consensus_queue.cc:260] T 83caee4530ca4cad846420cc4b7157e0 P 215645205d7e428f821c22ca54aab1ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb1729bd2424930bc5bebc205724683" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 39087 } } peers { permanent_uuid: "18513438818e4a07954992bc9a60c4e8" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 33619 } } peers { permanent_uuid: "215645205d7e428f821c22ca54aab1ee" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 40461 } }
I20260928 06:37:19.652593 3666 ts_tablet_manager.cc:1428] T 83caee4530ca4cad846420cc4b7157e0 P 215645205d7e428f821c22ca54aab1ee: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:37:19.652663 3386 heartbeater.cc:503] Master 127.16.157.126:39425 was elected leader, sending a full tablet report...
W20260928 06:37:19.746806 3518 tablet.cc:2369] T 83caee4530ca4cad846420cc4b7157e0 P fcb1729bd2424930bc5bebc205724683: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:37:19.805368 3669 raft_consensus.cc:482] T 83caee4530ca4cad846420cc4b7157e0 P 18513438818e4a07954992bc9a60c4e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:37:19.805493 3669 raft_consensus.cc:504] T 83caee4530ca4cad846420cc4b7157e0 P 18513438818e4a07954992bc9a60c4e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb1729bd2424930bc5bebc205724683" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 39087 } } peers { permanent_uuid: "18513438818e4a07954992bc9a60c4e8" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 33619 } } peers { permanent_uuid: "215645205d7e428f821c22ca54aab1ee" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 40461 } }
I20260928 06:37:19.805932 3669 leader_election.cc:288] T 83caee4530ca4cad846420cc4b7157e0 P 18513438818e4a07954992bc9a60c4e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fcb1729bd2424930bc5bebc205724683 (127.16.157.66:39087), 215645205d7e428f821c22ca54aab1ee (127.16.157.65:40461)
I20260928 06:37:19.808754 3661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:19.806210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:58825 (local address 127.16.157.65:40461)
0928 06:37:19.806280 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:19.806283 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:19.806291 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:37:19.806326 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:19.806327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:19.806347 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:19.806424 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:37:19.806586 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:19.807082 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:19.808021 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:19.808397 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:19.808485 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:19.808494 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:19.808641 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 06:37:19.808682 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 06:37:19.808763 3650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:19.806096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.65:40461 (local address 127.16.157.67:58825)
0928 06:37:19.806136 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:19.806149 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:19.806202 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:19.806423 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:19.806424 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:19.806425 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:37:19.806509 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:19.806511 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:19.807178 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:19.807179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:19.807922 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:19.807924 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:19.808482 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:19.808483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:19.808565 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:19.808568 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:37:19.808635 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 06:37:19.808679 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260928 06:37:19.808840 3660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:19.806188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:59687 (local address 127.16.157.66:39087)
0928 06:37:19.806306 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:19.806308 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:19.806383 (+ 75us) server_negotiation.cc:407] Connection header received
0928 06:37:19.806480 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:19.806481 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:19.806490 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:19.806546 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:37:19.806711 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:19.807177 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:19.808113 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:19.808478 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:19.808563 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:19.808571 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:19.808723 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 06:37:19.808765 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260928 06:37:19.808944 3671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:19.806101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.66:39087 (local address 127.16.157.67:59687)
0928 06:37:19.806300 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:19.806311 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:19.806400 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:19.806551 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:19.806552 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:19.806553 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:37:19.806628 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:19.806630 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:19.807261 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:19.807262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:19.808024 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:19.808026 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:19.808563 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:19.808564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:19.808643 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:19.808646 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:37:19.808717 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 06:37:19.808755 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":15,"thread_start_us":124,"threads_started":1}
I20260928 06:37:19.809170 3340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83caee4530ca4cad846420cc4b7157e0" candidate_uuid: "18513438818e4a07954992bc9a60c4e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "215645205d7e428f821c22ca54aab1ee" is_pre_election: true
I20260928 06:37:19.809309 3471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83caee4530ca4cad846420cc4b7157e0" candidate_uuid: "18513438818e4a07954992bc9a60c4e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcb1729bd2424930bc5bebc205724683" is_pre_election: true
I20260928 06:37:19.809355 3340 raft_consensus.cc:2454] T 83caee4530ca4cad846420cc4b7157e0 P 215645205d7e428f821c22ca54aab1ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18513438818e4a07954992bc9a60c4e8 in term 0.
I20260928 06:37:19.809474 3471 raft_consensus.cc:2454] T 83caee4530ca4cad846420cc4b7157e0 P fcb1729bd2424930bc5bebc205724683 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18513438818e4a07954992bc9a60c4e8 in term 0.
I20260928 06:37:19.809626 3535 leader_election.cc:302] T 83caee4530ca4cad846420cc4b7157e0 P 18513438818e4a07954992bc9a60c4e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18513438818e4a07954992bc9a60c4e8, 215645205d7e428f821c22ca54aab1ee; no voters:
I20260928 06:37:19.809801 3669 raft_consensus.cc:2781] T 83caee4530ca4cad846420cc4b7157e0 P 18513438818e4a07954992bc9a60c4e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:37:19.809849 3669 raft_consensus.cc:482] T 83caee4530ca4cad846420cc4b7157e0 P 18513438818e4a07954992bc9a60c4e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:37:19.809864 3669 raft_consensus.cc:3037] T 83caee4530ca4cad846420cc4b7157e0 P 18513438818e4a07954992bc9a60c4e8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:37:19.810547 3669 raft_consensus.cc:504] T 83caee4530ca4cad846420cc4b7157e0 P 18513438818e4a07954992bc9a60c4e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb1729bd2424930bc5bebc205724683" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 39087 } } peers { permanent_uuid: "18513438818e4a07954992bc9a60c4e8" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 33619 } } peers { permanent_uuid: "215645205d7e428f821c22ca54aab1ee" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 40461 } }
I20260928 06:37:19.810693 3669 leader_election.cc:288] T 83caee4530ca4cad846420cc4b7157e0 P 18513438818e4a07954992bc9a60c4e8 [CANDIDATE]: Term 1 election: Requested vote from peers fcb1729bd2424930bc5bebc205724683 (127.16.157.66:39087), 215645205d7e428f821c22ca54aab1ee (127.16.157.65:40461)
I20260928 06:37:19.810854 3471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83caee4530ca4cad846420cc4b7157e0" candidate_uuid: "18513438818e4a07954992bc9a60c4e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcb1729bd2424930bc5bebc205724683"
I20260928 06:37:19.810936 3471 raft_consensus.cc:3037] T 83caee4530ca4cad846420cc4b7157e0 P fcb1729bd2424930bc5bebc205724683 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:37:19.811813 3471 raft_consensus.cc:2454] T 83caee4530ca4cad846420cc4b7157e0 P fcb1729bd2424930bc5bebc205724683 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18513438818e4a07954992bc9a60c4e8 in term 1.
I20260928 06:37:19.811926 3340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83caee4530ca4cad846420cc4b7157e0" candidate_uuid: "18513438818e4a07954992bc9a60c4e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "215645205d7e428f821c22ca54aab1ee"
I20260928 06:37:19.812007 3538 leader_election.cc:302] T 83caee4530ca4cad846420cc4b7157e0 P 18513438818e4a07954992bc9a60c4e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18513438818e4a07954992bc9a60c4e8, fcb1729bd2424930bc5bebc205724683; no voters:
I20260928 06:37:19.812024 3340 raft_consensus.cc:3037] T 83caee4530ca4cad846420cc4b7157e0 P 215645205d7e428f821c22ca54aab1ee [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:37:19.812753 3340 raft_consensus.cc:2454] T 83caee4530ca4cad846420cc4b7157e0 P 215645205d7e428f821c22ca54aab1ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18513438818e4a07954992bc9a60c4e8 in term 1.
I20260928 06:37:19.812868 3669 raft_consensus.cc:2781] T 83caee4530ca4cad846420cc4b7157e0 P 18513438818e4a07954992bc9a60c4e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:37:19.813030 3669 raft_consensus.cc:686] T 83caee4530ca4cad846420cc4b7157e0 P 18513438818e4a07954992bc9a60c4e8 [term 1 LEADER]: Becoming Leader. State: Replica: 18513438818e4a07954992bc9a60c4e8, State: Running, Role: LEADER
I20260928 06:37:19.813134 3669 consensus_queue.cc:237] T 83caee4530ca4cad846420cc4b7157e0 P 18513438818e4a07954992bc9a60c4e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb1729bd2424930bc5bebc205724683" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 39087 } } peers { permanent_uuid: "18513438818e4a07954992bc9a60c4e8" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 33619 } } peers { permanent_uuid: "215645205d7e428f821c22ca54aab1ee" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 40461 } }
I20260928 06:37:19.814066 3197 catalog_manager.cc:5495] T 83caee4530ca4cad846420cc4b7157e0 P 18513438818e4a07954992bc9a60c4e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 18513438818e4a07954992bc9a60c4e8 (127.16.157.67). New cstate: current_term: 1 leader_uuid: "18513438818e4a07954992bc9a60c4e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb1729bd2424930bc5bebc205724683" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 39087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18513438818e4a07954992bc9a60c4e8" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 33619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "215645205d7e428f821c22ca54aab1ee" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 40461 } health_report { overall_health: UNKNOWN } } }
I20260928 06:37:19.822391 3659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:19.820190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58600 (local address 127.16.157.67:33619)
0928 06:37:19.820233 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:19.820235 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:19.820270 (+ 35us) server_negotiation.cc:407] Connection header received
0928 06:37:19.820301 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:19.820302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:19.820317 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:19.820370 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:37:19.820476 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:19.820938 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:19.821748 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:19.821935 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:19.821996 (+ 61us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:19.822118 (+ 122us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:37:19.822199 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:37:19.822266 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:19.822315 (+ 49us) server_negotiation.cc:299] Negotiation successful
0928 06:37:19.822323 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:37:19.824132 3471 raft_consensus.cc:1261] T 83caee4530ca4cad846420cc4b7157e0 P fcb1729bd2424930bc5bebc205724683 [term 1 FOLLOWER]: Refusing update from remote peer 18513438818e4a07954992bc9a60c4e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:37:19.824126 3340 raft_consensus.cc:1261] T 83caee4530ca4cad846420cc4b7157e0 P 215645205d7e428f821c22ca54aab1ee [term 1 FOLLOWER]: Refusing update from remote peer 18513438818e4a07954992bc9a60c4e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:37:19.824491 3669 consensus_queue.cc:1035] T 83caee4530ca4cad846420cc4b7157e0 P 18513438818e4a07954992bc9a60c4e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcb1729bd2424930bc5bebc205724683" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 39087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:37:19.824638 3672 consensus_queue.cc:1035] T 83caee4530ca4cad846420cc4b7157e0 P 18513438818e4a07954992bc9a60c4e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "215645205d7e428f821c22ca54aab1ee" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 40461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:37:19.829907 3197 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:37:19.830001 3197 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:37:19.830721 3197 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c8fd5e2a55e84a12a649de0b000b8884: Sending DeleteTablet for 3 replicas of tablet 83caee4530ca4cad846420cc4b7157e0
I20260928 06:37:19.831074 3582 tablet_service.cc:1514] Processing DeleteTablet for tablet 83caee4530ca4cad846420cc4b7157e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:37:19 UTC) from {username='slave'} at 127.0.0.1:58584
I20260928 06:37:19.831097 3320 tablet_service.cc:1514] Processing DeleteTablet for tablet 83caee4530ca4cad846420cc4b7157e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:37:19 UTC) from {username='slave'} at 127.0.0.1:54014
I20260928 06:37:19.831331 3686 tablet_replica.cc:330] T 83caee4530ca4cad846420cc4b7157e0 P 215645205d7e428f821c22ca54aab1ee: stopping tablet replica
I20260928 06:37:19.831416 3451 tablet_service.cc:1514] Processing DeleteTablet for tablet 83caee4530ca4cad846420cc4b7157e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:37:19 UTC) from {username='slave'} at 127.0.0.1:37210
I20260928 06:37:19.831529 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 3258
I20260928 06:37:19.831538 3686 raft_consensus.cc:2229] T 83caee4530ca4cad846420cc4b7157e0 P 215645205d7e428f821c22ca54aab1ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:37:19.831708 3686 raft_consensus.cc:2258] T 83caee4530ca4cad846420cc4b7157e0 P 215645205d7e428f821c22ca54aab1ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:37:19.831761 3687 tablet_replica.cc:330] T 83caee4530ca4cad846420cc4b7157e0 P fcb1729bd2424930bc5bebc205724683: stopping tablet replica
I20260928 06:37:19.831856 3687 raft_consensus.cc:2229] T 83caee4530ca4cad846420cc4b7157e0 P fcb1729bd2424930bc5bebc205724683 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:37:19.831959 3687 raft_consensus.cc:2258] T 83caee4530ca4cad846420cc4b7157e0 P fcb1729bd2424930bc5bebc205724683 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:37:19.832683 3687 ts_tablet_manager.cc:1905] T 83caee4530ca4cad846420cc4b7157e0 P fcb1729bd2424930bc5bebc205724683: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:37:19.834292 3687 ts_tablet_manager.cc:1918] T 83caee4530ca4cad846420cc4b7157e0 P fcb1729bd2424930bc5bebc205724683: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:37:19.834380 3687 log.cc:1196] T 83caee4530ca4cad846420cc4b7157e0 P fcb1729bd2424930bc5bebc205724683: Deleting WAL directory at /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577196946045-17013-0/minicluster-data/ts-1/wal/wals/83caee4530ca4cad846420cc4b7157e0
I20260928 06:37:19.834635 3687 ts_tablet_manager.cc:1939] T 83caee4530ca4cad846420cc4b7157e0 P fcb1729bd2424930bc5bebc205724683: Deleting consensus metadata
I20260928 06:37:19.835054 3185 catalog_manager.cc:4841] TS fcb1729bd2424930bc5bebc205724683 (127.16.157.66:39087): tablet 83caee4530ca4cad846420cc4b7157e0 (table test-table [id=b9c96980124441b6832710b0c3f2189c]) successfully deleted
I20260928 06:37:19.837785 3685 tablet_replica.cc:330] T 83caee4530ca4cad846420cc4b7157e0 P 18513438818e4a07954992bc9a60c4e8: stopping tablet replica
I20260928 06:37:19.837917 3685 raft_consensus.cc:2229] T 83caee4530ca4cad846420cc4b7157e0 P 18513438818e4a07954992bc9a60c4e8 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:37:19.838088 3685 raft_consensus.cc:2258] T 83caee4530ca4cad846420cc4b7157e0 P 18513438818e4a07954992bc9a60c4e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:37:19.838575 3685 ts_tablet_manager.cc:1905] T 83caee4530ca4cad846420cc4b7157e0 P 18513438818e4a07954992bc9a60c4e8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 06:37:19.839946 3182 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.157.65:40461 (error 108)
I20260928 06:37:19.840072 3685 ts_tablet_manager.cc:1918] T 83caee4530ca4cad846420cc4b7157e0 P 18513438818e4a07954992bc9a60c4e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:37:19.840168 3685 log.cc:1196] T 83caee4530ca4cad846420cc4b7157e0 P 18513438818e4a07954992bc9a60c4e8: Deleting WAL directory at /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577196946045-17013-0/minicluster-data/ts-2/wal/wals/83caee4530ca4cad846420cc4b7157e0
I20260928 06:37:19.840466 3685 ts_tablet_manager.cc:1939] T 83caee4530ca4cad846420cc4b7157e0 P 18513438818e4a07954992bc9a60c4e8: Deleting consensus metadata
I20260928 06:37:19.840590 3657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:19.840410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.65:40461 (local address 127.0.0.1:54030)
0928 06:37:19.840478 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:19.840528 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.65:40461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 06:37:19.840651 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 3389
I20260928 06:37:19.840919 3184 catalog_manager.cc:4841] TS 18513438818e4a07954992bc9a60c4e8 (127.16.157.67:33619): tablet 83caee4530ca4cad846420cc4b7157e0 (table test-table [id=b9c96980124441b6832710b0c3f2189c]) successfully deleted
W20260928 06:37:19.842602 3182 catalog_manager.cc:4584] TS 215645205d7e428f821c22ca54aab1ee (127.16.157.65:40461): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 83caee4530ca4cad846420cc4b7157e0: Network error: Client connection negotiation failed: client connection to 127.16.157.65:40461: connect: Connection refused (error 111)
I20260928 06:37:19.848220 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 3520
I20260928 06:37:19.853348 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 3167
2026-09-28T06:37:19Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (793 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:37:19 dist-test-slave-rrj5 krb5kdc[3691](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:37:19 dist-test-slave-rrj5 krb5kdc[3691](info): set up 2 sockets
Sep 28 06:37:19 dist-test-slave-rrj5 krb5kdc[3691](info): commencing operation
krb5kdc: starting...
W20260928 06:37:20.910595 17013 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Sep 28 06:37:20 dist-test-slave-rrj5 krb5kdc[3691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:37:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:37:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.157.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab.
Entry for principal kudu/127.16.157.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab.
I20260928 06:37:21.033424 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:39847
--webserver_interface=127.16.157.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=kudu/127.16.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:35565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.157.126:39847
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:37:21.116717 3704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:21.116986 3704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:21.117024 3704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:21.118213 3704 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:37:21.118288 3704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:21.118314 3704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:21.118336 3704 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:37:21.118366 3704 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:37:21.119837 3704 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:35565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.157.126:39847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=kudu/127.16.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:39847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:37:21.120136 3704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:21.120374 3704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:21.125336 3709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:21.125319 3710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:21.125305 3712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:21.125617 3704 server_base.cc:1034] running on GCE node
I20260928 06:37:21.126014 3704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:21.126320 3704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:21.127473 3704 hybrid_clock.cc:648] HybridClock initialized: now 1790577441127454 us; error 41 us; skew 500 ppm
Sep 28 06:37:21 dist-test-slave-rrj5 krb5kdc[3691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577441, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:21.130048 3704 init.cc:377] Logged in from keytab as kudu/127.16.157.126@KRBTEST.COM (short username kudu)
I20260928 06:37:21.130535 3704 webserver.cc:460] Webserver started at http://127.16.157.126:42671/ using document root <none> and password file <none>
I20260928 06:37:21.130788 3704 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:21.130859 3704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:21.131006 3704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:37:21.132125 3704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/minicluster-data/master-0/data/instance:
uuid: "1ec7d17a8c534533a2aee104948074de"
format_stamp: "Formatted at 2026-09-28 06:37:21 on dist-test-slave-rrj5"
server_key: "50f06c013c0625f19a85e61007fddd0c"
server_key_iv: "dbf8d4d39322bdba9f07b765ea638664"
server_key_version: "encryptionkey@0"
I20260928 06:37:21.132499 3704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/minicluster-data/master-0/wal/instance:
uuid: "1ec7d17a8c534533a2aee104948074de"
format_stamp: "Formatted at 2026-09-28 06:37:21 on dist-test-slave-rrj5"
server_key: "50f06c013c0625f19a85e61007fddd0c"
server_key_iv: "dbf8d4d39322bdba9f07b765ea638664"
server_key_version: "encryptionkey@0"
I20260928 06:37:21.133930 3704 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 06:37:21.134733 3719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:21.134919 3704 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:37:21.135023 3704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/minicluster-data/master-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/minicluster-data/master-0/wal
uuid: "1ec7d17a8c534533a2aee104948074de"
format_stamp: "Formatted at 2026-09-28 06:37:21 on dist-test-slave-rrj5"
server_key: "50f06c013c0625f19a85e61007fddd0c"
server_key_iv: "dbf8d4d39322bdba9f07b765ea638664"
server_key_version: "encryptionkey@0"
I20260928 06:37:21.135113 3704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:21.156819 3704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:21.159392 3704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:21.159570 3704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:21.163879 3704 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.126:39847
I20260928 06:37:21.163921 3771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.126:39847 every 8 connection(s)
I20260928 06:37:21.164306 3704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
I20260928 06:37:21.165194 3772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:37:21.167594 3772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ec7d17a8c534533a2aee104948074de: Bootstrap starting.
I20260928 06:37:21.168215 3772 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ec7d17a8c534533a2aee104948074de: Neither blocks nor log segments found. Creating new log.
I20260928 06:37:21.168537 3772 log.cc:824] T 00000000000000000000000000000000 P 1ec7d17a8c534533a2aee104948074de: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:21.168900 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 3704
I20260928 06:37:21.169020 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/minicluster-data/master-0/wal/instance
I20260928 06:37:21.169239 17013 external_mini_cluster.cc:1428] Setting key 7ada462b162c0fdbb0afcc3a2dd7f726
I20260928 06:37:21.169473 3772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ec7d17a8c534533a2aee104948074de: No bootstrap required, opened a new log
I20260928 06:37:21.171921 3772 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ec7d17a8c534533a2aee104948074de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ec7d17a8c534533a2aee104948074de" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 39847 } }
I20260928 06:37:21.172109 3772 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ec7d17a8c534533a2aee104948074de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:37:21.172168 3772 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ec7d17a8c534533a2aee104948074de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ec7d17a8c534533a2aee104948074de, State: Initialized, Role: FOLLOWER
I20260928 06:37:21.172322 3772 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ec7d17a8c534533a2aee104948074de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ec7d17a8c534533a2aee104948074de" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 39847 } }
I20260928 06:37:21.172432 3772 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ec7d17a8c534533a2aee104948074de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:37:21.172465 3772 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ec7d17a8c534533a2aee104948074de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:37:21.172509 3772 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ec7d17a8c534533a2aee104948074de [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:37:21.173239 3772 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ec7d17a8c534533a2aee104948074de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ec7d17a8c534533a2aee104948074de" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 39847 } }
I20260928 06:37:21.173365 3772 leader_election.cc:302] T 00000000000000000000000000000000 P 1ec7d17a8c534533a2aee104948074de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ec7d17a8c534533a2aee104948074de; no voters:
I20260928 06:37:21.173522 3772 leader_election.cc:288] T 00000000000000000000000000000000 P 1ec7d17a8c534533a2aee104948074de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:37:21.173565 3777 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ec7d17a8c534533a2aee104948074de [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:37:21.173779 3777 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ec7d17a8c534533a2aee104948074de [term 1 LEADER]: Becoming Leader. State: Replica: 1ec7d17a8c534533a2aee104948074de, State: Running, Role: LEADER
I20260928 06:37:21.173810 3772 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ec7d17a8c534533a2aee104948074de [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:37:21.173869 3777 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ec7d17a8c534533a2aee104948074de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ec7d17a8c534533a2aee104948074de" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 39847 } }
I20260928 06:37:21.174208 3777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ec7d17a8c534533a2aee104948074de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ec7d17a8c534533a2aee104948074de. Latest consensus state: current_term: 1 leader_uuid: "1ec7d17a8c534533a2aee104948074de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ec7d17a8c534533a2aee104948074de" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 39847 } } }
I20260928 06:37:21.174307 3777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ec7d17a8c534533a2aee104948074de [sys.catalog]: This master's current role is: LEADER
I20260928 06:37:21.174219 3778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ec7d17a8c534533a2aee104948074de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ec7d17a8c534533a2aee104948074de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ec7d17a8c534533a2aee104948074de" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 39847 } } }
I20260928 06:37:21.174408 3778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ec7d17a8c534533a2aee104948074de [sys.catalog]: This master's current role is: LEADER
I20260928 06:37:21.174535 3784 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:37:21.175282 3784 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 06:37:21 dist-test-slave-rrj5 krb5kdc[3691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:37:21.178241 3784 catalog_manager.cc:1347] Generated new cluster ID: 192e78bb806c47aab199a4c026002d87
I20260928 06:37:21.178289 3784 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:37:21.182483 3776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:21.169796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38562 (local address 127.16.157.126:39847)
0928 06:37:21.170063 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:21.170074 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:21.170115 (+ 41us) server_negotiation.cc:407] Connection header received
0928 06:37:21.171063 (+ 948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:21.171073 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:21.171142 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:21.171292 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:21.172010 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:21.173780 (+ 1770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:21.175026 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:21.175634 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:21.175742 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:21.178758 (+ 3016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:21.178762 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:21.178762 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:21.178773 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:21.180755 (+ 1982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:21.181174 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:21.181177 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:21.181178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:21.181281 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:21.181502 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:21.181503 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:21.181503 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:21.181812 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:21.181965 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:21.182141 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 06:37:21.182289 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":101,"threads_started":1}
I20260928 06:37:21.183145 3784 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:37:21.183661 3784 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:37:21.191826 3784 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ec7d17a8c534533a2aee104948074de: Generated new TSK 0
I20260928 06:37:21.192042 3784 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.157.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab.
Entry for principal kudu/127.16.157.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab.
I20260928 06:37:21.220856 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.65:0
--local_ip_for_outbound_sockets=127.16.157.65
--webserver_interface=127.16.157.65
--webserver_port=0
--tserver_master_addrs=127.16.157.126:39847
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab
--principal=kudu/127.16.157.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:35565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:37:21.304715 3798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:21.305011 3798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:21.305073 3798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:21.306371 3798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:21.306442 3798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:21.306488 3798 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.65
I20260928 06:37:21.308010 3798 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:35565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.65.keytab
--principal=kudu/127.16.157.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.157.65
--webserver_port=0
--tserver_master_addrs=127.16.157.126:39847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3798
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.65
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:37:21.308291 3798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:21.308535 3798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:21.313753 3803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:21.313772 3804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:21.313884 3806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:21.313968 3798 server_base.cc:1034] running on GCE node
I20260928 06:37:21.314391 3798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:21.314692 3798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:21.315845 3798 hybrid_clock.cc:648] HybridClock initialized: now 1790577441315829 us; error 42 us; skew 500 ppm
Sep 28 06:37:21 dist-test-slave-rrj5 krb5kdc[3691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577441, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:21.318359 3798 init.cc:377] Logged in from keytab as kudu/127.16.157.65@KRBTEST.COM (short username kudu)
I20260928 06:37:21.318842 3798 webserver.cc:460] Webserver started at http://127.16.157.65:35469/ using document root <none> and password file <none>
I20260928 06:37:21.319082 3798 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:21.319151 3798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:21.319306 3798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:37:21.320389 3798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/minicluster-data/ts-0/data/instance:
uuid: "f5bc49ea35e445dfb53e181c24e167f1"
format_stamp: "Formatted at 2026-09-28 06:37:21 on dist-test-slave-rrj5"
server_key: "c6bb59145451faee0e0fb3f0edb6b9b3"
server_key_iv: "d97329464de9ba7e40562233b9d46917"
server_key_version: "encryptionkey@0"
I20260928 06:37:21.320765 3798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/minicluster-data/ts-0/wal/instance:
uuid: "f5bc49ea35e445dfb53e181c24e167f1"
format_stamp: "Formatted at 2026-09-28 06:37:21 on dist-test-slave-rrj5"
server_key: "c6bb59145451faee0e0fb3f0edb6b9b3"
server_key_iv: "d97329464de9ba7e40562233b9d46917"
server_key_version: "encryptionkey@0"
I20260928 06:37:21.322219 3798 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 06:37:21.322973 3813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:21.323141 3798 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:37:21.323246 3798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/minicluster-data/ts-0/wal
uuid: "f5bc49ea35e445dfb53e181c24e167f1"
format_stamp: "Formatted at 2026-09-28 06:37:21 on dist-test-slave-rrj5"
server_key: "c6bb59145451faee0e0fb3f0edb6b9b3"
server_key_iv: "d97329464de9ba7e40562233b9d46917"
server_key_version: "encryptionkey@0"
I20260928 06:37:21.323320 3798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:21.339178 3798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:21.340997 3798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:21.341172 3798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:21.341406 3798 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:37:21.341818 3798 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:37:21.341873 3798 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:21.341918 3798 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:37:21.341948 3798 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:21.348062 3798 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.65:42393
I20260928 06:37:21.348105 3926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.65:42393 every 8 connection(s)
I20260928 06:37:21.348430 3798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
Sep 28 06:37:21 dist-test-slave-rrj5 krb5kdc[3691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577441, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.65@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:37:21.356702 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 3798
I20260928 06:37:21.356835 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/minicluster-data/ts-0/wal/instance
I20260928 06:37:21.357059 17013 external_mini_cluster.cc:1428] Setting key ec91733e7e7bd0c4242599dac79c9399
I20260928 06:37:21.358121 3776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:21.349328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:42509 (local address 127.16.157.126:39847)
0928 06:37:21.349415 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:21.349417 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:21.349637 (+ 220us) server_negotiation.cc:407] Connection header received
0928 06:37:21.350186 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:21.350187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:21.350199 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:21.350259 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:21.351241 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:21.351778 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:21.352802 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:21.353036 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:21.353119 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:21.355319 (+ 2200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:21.355320 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:21.355321 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:21.355327 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:21.356533 (+ 1206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:21.357050 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:21.357051 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:21.357052 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:21.357123 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:21.357374 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:21.357375 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:21.357376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:21.357543 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:21.357667 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:21.357979 (+ 312us) server_negotiation.cc:299] Negotiation successful
0928 06:37:21.358058 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 06:37:21.358443 3929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:21.349282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:39847 (local address 127.16.157.65:42509)
0928 06:37:21.349480 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:21.349527 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:21.350078 (+ 551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:21.350330 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:21.350334 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:21.350921 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:37:21.351133 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:21.351138 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:21.351882 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:21.351884 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:21.352706 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:21.352708 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:21.353139 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:21.353140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:21.353268 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:21.353803 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:37:21.353811 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:37:21.355227 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:37:21.356630 (+ 1403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:21.356640 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:21.356642 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:21.356977 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:21.357207 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:21.357207 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:21.357208 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:21.357325 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:21.357670 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:37:21.357676 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:37:21.357831 (+ 155us) client_negotiation.cc:769] Sending connection context
0928 06:37:21.357958 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 06:37:21.358156 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":99,"threads_started":1}
I20260928 06:37:21.358958 3927 heartbeater.cc:347] Connected to a master server at 127.16.157.126:39847
I20260928 06:37:21.359097 3927 heartbeater.cc:464] Registering TS with master...
I20260928 06:37:21.359301 3927 heartbeater.cc:511] Master 127.16.157.126:39847 requested a full tablet report, sending...
I20260928 06:37:21.359865 3736 ts_manager.cc:194] Registered new tserver with Master: f5bc49ea35e445dfb53e181c24e167f1 (127.16.157.65:42393)
I20260928 06:37:21.360632 3736 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.157.65@KRBTEST.COM'} at 127.16.157.65:42509
WARNING: no policy specified for kudu/127.16.157.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab.
Entry for principal kudu/127.16.157.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab.
I20260928 06:37:21.384259 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.66:0
--local_ip_for_outbound_sockets=127.16.157.66
--webserver_interface=127.16.157.66
--webserver_port=0
--tserver_master_addrs=127.16.157.126:39847
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab
--principal=kudu/127.16.157.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:35565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:37:21.466594 3932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:21.466811 3932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:21.466836 3932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:21.467983 3932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:21.468034 3932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:21.468091 3932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.66
I20260928 06:37:21.469633 3932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:35565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.66.keytab
--principal=kudu/127.16.157.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.157.66
--webserver_port=0
--tserver_master_addrs=127.16.157.126:39847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.66
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:37:21.469875 3932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:21.470106 3932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:21.475338 3940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:21.475330 3938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:21.475329 3937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:21.475478 3932 server_base.cc:1034] running on GCE node
I20260928 06:37:21.475768 3932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:21.476013 3932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:21.477182 3932 hybrid_clock.cc:648] HybridClock initialized: now 1790577441477171 us; error 27 us; skew 500 ppm
Sep 28 06:37:21 dist-test-slave-rrj5 krb5kdc[3691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577441, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:21.479598 3932 init.cc:377] Logged in from keytab as kudu/127.16.157.66@KRBTEST.COM (short username kudu)
I20260928 06:37:21.480067 3932 webserver.cc:460] Webserver started at http://127.16.157.66:42793/ using document root <none> and password file <none>
I20260928 06:37:21.480286 3932 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:21.480352 3932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:21.480489 3932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:37:21.481506 3932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/minicluster-data/ts-1/data/instance:
uuid: "1ed91ab0a8e54c48ba26ae140d3d696d"
format_stamp: "Formatted at 2026-09-28 06:37:21 on dist-test-slave-rrj5"
server_key: "6039249bd53fc4371d2be549ddb99790"
server_key_iv: "1ed4af76fc406f0a43dfc727e95e8fb9"
server_key_version: "encryptionkey@0"
I20260928 06:37:21.481947 3932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/minicluster-data/ts-1/wal/instance:
uuid: "1ed91ab0a8e54c48ba26ae140d3d696d"
format_stamp: "Formatted at 2026-09-28 06:37:21 on dist-test-slave-rrj5"
server_key: "6039249bd53fc4371d2be549ddb99790"
server_key_iv: "1ed4af76fc406f0a43dfc727e95e8fb9"
server_key_version: "encryptionkey@0"
I20260928 06:37:21.483249 3932 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:37:21.483965 3947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:21.484143 3932 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:37:21.484239 3932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/minicluster-data/ts-1/wal
uuid: "1ed91ab0a8e54c48ba26ae140d3d696d"
format_stamp: "Formatted at 2026-09-28 06:37:21 on dist-test-slave-rrj5"
server_key: "6039249bd53fc4371d2be549ddb99790"
server_key_iv: "1ed4af76fc406f0a43dfc727e95e8fb9"
server_key_version: "encryptionkey@0"
I20260928 06:37:21.484320 3932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:21.513151 3932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:21.515084 3932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:21.515239 3932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:21.515470 3932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:37:21.515789 3932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:37:21.515842 3932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:21.515884 3932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:37:21.515924 3932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:21.522101 3932 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.66:46705
I20260928 06:37:21.522164 4060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.66:46705 every 8 connection(s)
I20260928 06:37:21.522501 3932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
Sep 28 06:37:21 dist-test-slave-rrj5 krb5kdc[3691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577441, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.66@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:37:21.529673 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 3932
I20260928 06:37:21.529799 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/minicluster-data/ts-1/wal/instance
I20260928 06:37:21.529996 17013 external_mini_cluster.cc:1428] Setting key 4a130eb1ff15ee1d3701cf63f793bdba
I20260928 06:37:21.531342 3776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:21.523190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.66:35071 (local address 127.16.157.126:39847)
0928 06:37:21.523234 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:21.523236 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:21.523424 (+ 188us) server_negotiation.cc:407] Connection header received
0928 06:37:21.523862 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:21.523863 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:21.523872 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:21.523925 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:21.524771 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:21.525284 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:21.526373 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:21.526575 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:21.526629 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:21.528560 (+ 1931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:21.528561 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:21.528561 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:21.528567 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:21.529797 (+ 1230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:21.530267 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:21.530268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:21.530268 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:21.530334 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:21.530627 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:21.530628 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:21.530628 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:21.530765 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:21.530841 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:21.531100 (+ 259us) server_negotiation.cc:299] Negotiation successful
0928 06:37:21.531196 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:37:21.531365 4063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:21.523149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:39847 (local address 127.16.157.66:35071)
0928 06:37:21.523300 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:21.523337 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:21.523787 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:21.523973 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:21.523976 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:21.524505 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:37:21.524699 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:21.524704 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:21.525372 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:21.525373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:21.526297 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:21.526300 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:21.526647 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:21.526647 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:21.526738 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:21.527209 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:37:21.527217 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:37:21.528476 (+ 1259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:37:21.529898 (+ 1422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:21.529905 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:21.529907 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:21.530175 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:21.530407 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:21.530408 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:21.530408 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:21.530545 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:21.530837 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:37:21.530843 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:37:21.530950 (+ 107us) client_negotiation.cc:769] Sending connection context
0928 06:37:21.531073 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 06:37:21.531213 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":83,"threads_started":1}
I20260928 06:37:21.532544 4061 heartbeater.cc:347] Connected to a master server at 127.16.157.126:39847
I20260928 06:37:21.532626 4061 heartbeater.cc:464] Registering TS with master...
I20260928 06:37:21.532816 4061 heartbeater.cc:511] Master 127.16.157.126:39847 requested a full tablet report, sending...
I20260928 06:37:21.533202 3736 ts_manager.cc:194] Registered new tserver with Master: 1ed91ab0a8e54c48ba26ae140d3d696d (127.16.157.66:46705)
I20260928 06:37:21.533648 3736 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.157.66@KRBTEST.COM'} at 127.16.157.66:35071
WARNING: no policy specified for kudu/127.16.157.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab.
Entry for principal kudu/127.16.157.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab.
I20260928 06:37:21.556165 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.67:0
--local_ip_for_outbound_sockets=127.16.157.67
--webserver_interface=127.16.157.67
--webserver_port=0
--tserver_master_addrs=127.16.157.126:39847
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab
--principal=kudu/127.16.157.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:35565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:37:21.637672 4066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:21.637892 4066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:21.637930 4066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:21.639083 4066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:21.639153 4066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:21.639199 4066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.67
I20260928 06:37:21.640643 4066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:35565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.67.keytab
--principal=kudu/127.16.157.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.157.67
--webserver_port=0
--tserver_master_addrs=127.16.157.126:39847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.67
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:37:21.640888 4066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:21.641103 4066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:21.645975 4071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:21.645983 4074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:21.645984 4072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:21.646239 4066 server_base.cc:1034] running on GCE node
I20260928 06:37:21.646420 4066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:21.646606 4066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:21.647754 4066 hybrid_clock.cc:648] HybridClock initialized: now 1790577441647736 us; error 34 us; skew 500 ppm
Sep 28 06:37:21 dist-test-slave-rrj5 krb5kdc[3691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577441, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:21.650260 4066 init.cc:377] Logged in from keytab as kudu/127.16.157.67@KRBTEST.COM (short username kudu)
I20260928 06:37:21.650795 4066 webserver.cc:460] Webserver started at http://127.16.157.67:34923/ using document root <none> and password file <none>
I20260928 06:37:21.651005 4066 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:21.651049 4066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:21.651155 4066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:37:21.652166 4066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/minicluster-data/ts-2/data/instance:
uuid: "f0e81cf01a4a45678a109bfcf159bd30"
format_stamp: "Formatted at 2026-09-28 06:37:21 on dist-test-slave-rrj5"
server_key: "35594a0a4531e4c334047550c5303eea"
server_key_iv: "50c1338cac018aa514462135089eef04"
server_key_version: "encryptionkey@0"
I20260928 06:37:21.652455 4066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/minicluster-data/ts-2/wal/instance:
uuid: "f0e81cf01a4a45678a109bfcf159bd30"
format_stamp: "Formatted at 2026-09-28 06:37:21 on dist-test-slave-rrj5"
server_key: "35594a0a4531e4c334047550c5303eea"
server_key_iv: "50c1338cac018aa514462135089eef04"
server_key_version: "encryptionkey@0"
I20260928 06:37:21.653704 4066 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:37:21.654398 4081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:21.654569 4066 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:37:21.654675 4066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/minicluster-data/ts-2/wal
uuid: "f0e81cf01a4a45678a109bfcf159bd30"
format_stamp: "Formatted at 2026-09-28 06:37:21 on dist-test-slave-rrj5"
server_key: "35594a0a4531e4c334047550c5303eea"
server_key_iv: "50c1338cac018aa514462135089eef04"
server_key_version: "encryptionkey@0"
I20260928 06:37:21.654750 4066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:21.670051 4066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:21.672467 4066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:21.672624 4066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:21.672885 4066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:37:21.673205 4066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:37:21.673260 4066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:21.673301 4066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:37:21.673329 4066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:21.679562 4066 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.67:36245
I20260928 06:37:21.679603 4194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.67:36245 every 8 connection(s)
I20260928 06:37:21.679930 4066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
I20260928 06:37:21.681061 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 4066
I20260928 06:37:21.681188 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/minicluster-data/ts-2/wal/instance
I20260928 06:37:21.681372 17013 external_mini_cluster.cc:1428] Setting key 1f7360206f1bcee91e2e5f7aef1a14c0
Sep 28 06:37:21 dist-test-slave-rrj5 krb5kdc[3691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577441, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.67@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:37:21.689318 3776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:21.680674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:40867 (local address 127.16.157.126:39847)
0928 06:37:21.680749 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:21.680751 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:21.680961 (+ 210us) server_negotiation.cc:407] Connection header received
0928 06:37:21.681731 (+ 770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:21.681733 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:21.681747 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:21.681811 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:21.682909 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:21.683413 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:21.684369 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:21.684559 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:21.684609 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:21.686704 (+ 2095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:21.686705 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:21.686706 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:21.686713 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:21.687904 (+ 1191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:21.688343 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:21.688344 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:21.688345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:21.688405 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:21.688665 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:21.688666 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:21.688667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:21.688787 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:21.688867 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:21.689144 (+ 277us) server_negotiation.cc:299] Negotiation successful
0928 06:37:21.689228 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 06:37:21.689527 4197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:21.680649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:39847 (local address 127.16.157.67:40867)
0928 06:37:21.680820 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:21.680864 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:21.681609 (+ 745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:21.681892 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:21.681896 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:21.682621 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:37:21.682825 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:21.682831 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:21.683492 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:21.683493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:21.684297 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:21.684300 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:21.684629 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:21.684630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:21.684732 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:21.685159 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:37:21.685167 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:37:21.686590 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:37:21.687991 (+ 1401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:21.687998 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:21.688001 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:21.688250 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:21.688480 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:21.688481 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:21.688481 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:21.688597 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:21.688859 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:37:21.688866 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:37:21.689008 (+ 142us) client_negotiation.cc:769] Sending connection context
0928 06:37:21.689122 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 06:37:21.689280 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":98,"threads_started":1}
I20260928 06:37:21.690013 4195 heartbeater.cc:347] Connected to a master server at 127.16.157.126:39847
I20260928 06:37:21.690102 4195 heartbeater.cc:464] Registering TS with master...
I20260928 06:37:21.690239 4195 heartbeater.cc:511] Master 127.16.157.126:39847 requested a full tablet report, sending...
I20260928 06:37:21.690569 3736 ts_manager.cc:194] Registered new tserver with Master: f0e81cf01a4a45678a109bfcf159bd30 (127.16.157.67:36245)
I20260928 06:37:21.690943 3736 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.157.67@KRBTEST.COM'} at 127.16.157.67:40867
I20260928 06:37:21.693096 17013 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:37:21.699702 3776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:21.694393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38572 (local address 127.16.157.126:39847)
0928 06:37:21.694479 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:21.694480 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:21.694516 (+ 36us) server_negotiation.cc:407] Connection header received
0928 06:37:21.694573 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:21.694573 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:21.694580 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:21.694625 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:21.695132 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:21.695588 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:21.696246 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:21.696418 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:21.696465 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:21.697513 (+ 1048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:21.697514 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:21.697514 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:21.697520 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:21.698710 (+ 1190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:21.699028 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:21.699029 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:21.699029 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:21.699076 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:21.699246 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:21.699246 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:21.699246 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:21.699390 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:21.699461 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:21.699562 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 06:37:21.699618 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 06:37:21.701282 3736 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38572:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:37:21.702906 3736 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:37:21.709613 4207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:21.705460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.67:36245 (local address 127.0.0.1:53724)
0928 06:37:21.705963 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:21.705972 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:21.706046 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:21.706295 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:21.706296 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:21.706297 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:37:21.706332 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:21.706333 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:21.707208 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:21.707209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:21.708498 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:21.708501 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:21.709254 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:21.709255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:21.709338 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:21.709352 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 06:37:21.709428 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 06:37:21.709499 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":71,"threads_started":1}
I20260928 06:37:21.709684 4209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:21.705287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.65:42393 (local address 127.0.0.1:42960)
0928 06:37:21.705616 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:21.705627 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:21.705686 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:21.705925 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:21.705928 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:21.705930 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:37:21.705999 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:21.706000 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:21.707540 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:21.707541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:21.708476 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:21.708491 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:21.709377 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:21.709378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:21.709442 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:21.709445 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:37:21.709510 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 06:37:21.709544 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":98,"threads_started":1}
I20260928 06:37:21.709877 4206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:21.705262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.66:46705 (local address 127.0.0.1:36156)
0928 06:37:21.705406 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:21.705451 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:21.705537 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:21.706068 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:21.706069 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:21.706071 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:37:21.706138 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:21.706140 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:21.707485 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:21.707486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:21.708608 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:21.708610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:21.709464 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:21.709466 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:21.709533 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:21.709537 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:37:21.709587 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 06:37:21.709627 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":1,"thread_start_us":58,"threads_started":1}
I20260928 06:37:21.709904 4208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:21.705371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36156 (local address 127.16.157.66:46705)
0928 06:37:21.705890 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:21.705893 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:21.705902 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:37:21.705940 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:21.705942 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:21.705956 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:21.706057 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:37:21.706227 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:21.707375 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:21.708694 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:21.709371 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:21.709672 (+ 301us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:21.709752 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:21.709782 (+ 30us) server_negotiation.cc:299] Negotiation successful
0928 06:37:21.709829 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":76,"threads_started":1}
I20260928 06:37:21.709988 4211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:21.705676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53724 (local address 127.16.157.67:36245)
0928 06:37:21.705824 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:21.705829 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:21.706125 (+ 296us) server_negotiation.cc:407] Connection header received
0928 06:37:21.706178 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:21.706180 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:21.706197 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:21.706298 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:37:21.706387 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:21.707078 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:21.708616 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:21.709137 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:21.709251 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:21.709314 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:21.709836 (+ 522us) server_negotiation.cc:299] Negotiation successful
0928 06:37:21.709886 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":82,"threads_started":1}
I20260928 06:37:21.710407 4210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:21.705482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42960 (local address 127.16.157.65:42393)
0928 06:37:21.705754 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:21.705758 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:21.705767 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:37:21.705801 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:21.705802 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:21.705815 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:21.705880 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:37:21.706375 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:21.707455 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:21.708575 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:21.709302 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:21.709380 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:21.709442 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:21.709525 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 06:37:21.709601 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":127,"threads_started":1}
I20260928 06:37:21.710626 3995 tablet_service.cc:1467] Processing CreateTablet for tablet 6dc698d297394664858732a519f2ed52 (DEFAULT_TABLE table=test-table [id=ef1ff00ce2974f0cb851f0785d56a734]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:37:21.710626 4129 tablet_service.cc:1467] Processing CreateTablet for tablet 6dc698d297394664858732a519f2ed52 (DEFAULT_TABLE table=test-table [id=ef1ff00ce2974f0cb851f0785d56a734]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:37:21.710988 3995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dc698d297394664858732a519f2ed52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:37:21.711000 4129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dc698d297394664858732a519f2ed52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:37:21.711113 3861 tablet_service.cc:1467] Processing CreateTablet for tablet 6dc698d297394664858732a519f2ed52 (DEFAULT_TABLE table=test-table [id=ef1ff00ce2974f0cb851f0785d56a734]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:37:21.711452 3861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dc698d297394664858732a519f2ed52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:37:21.714131 4212 tablet_bootstrap.cc:492] T 6dc698d297394664858732a519f2ed52 P f5bc49ea35e445dfb53e181c24e167f1: Bootstrap starting.
I20260928 06:37:21.714555 4213 tablet_bootstrap.cc:492] T 6dc698d297394664858732a519f2ed52 P 1ed91ab0a8e54c48ba26ae140d3d696d: Bootstrap starting.
I20260928 06:37:21.714548 4214 tablet_bootstrap.cc:492] T 6dc698d297394664858732a519f2ed52 P f0e81cf01a4a45678a109bfcf159bd30: Bootstrap starting.
I20260928 06:37:21.714943 4212 tablet_bootstrap.cc:654] T 6dc698d297394664858732a519f2ed52 P f5bc49ea35e445dfb53e181c24e167f1: Neither blocks nor log segments found. Creating new log.
I20260928 06:37:21.715255 4213 tablet_bootstrap.cc:654] T 6dc698d297394664858732a519f2ed52 P 1ed91ab0a8e54c48ba26ae140d3d696d: Neither blocks nor log segments found. Creating new log.
I20260928 06:37:21.715255 4214 tablet_bootstrap.cc:654] T 6dc698d297394664858732a519f2ed52 P f0e81cf01a4a45678a109bfcf159bd30: Neither blocks nor log segments found. Creating new log.
I20260928 06:37:21.715564 4212 log.cc:824] T 6dc698d297394664858732a519f2ed52 P f5bc49ea35e445dfb53e181c24e167f1: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:21.715910 4214 log.cc:824] T 6dc698d297394664858732a519f2ed52 P f0e81cf01a4a45678a109bfcf159bd30: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:21.715924 4213 log.cc:824] T 6dc698d297394664858732a519f2ed52 P 1ed91ab0a8e54c48ba26ae140d3d696d: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:21.716385 4212 tablet_bootstrap.cc:492] T 6dc698d297394664858732a519f2ed52 P f5bc49ea35e445dfb53e181c24e167f1: No bootstrap required, opened a new log
I20260928 06:37:21.716462 4212 ts_tablet_manager.cc:1397] T 6dc698d297394664858732a519f2ed52 P f5bc49ea35e445dfb53e181c24e167f1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:37:21.716735 4214 tablet_bootstrap.cc:492] T 6dc698d297394664858732a519f2ed52 P f0e81cf01a4a45678a109bfcf159bd30: No bootstrap required, opened a new log
I20260928 06:37:21.716809 4214 ts_tablet_manager.cc:1397] T 6dc698d297394664858732a519f2ed52 P f0e81cf01a4a45678a109bfcf159bd30: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:37:21.716980 4213 tablet_bootstrap.cc:492] T 6dc698d297394664858732a519f2ed52 P 1ed91ab0a8e54c48ba26ae140d3d696d: No bootstrap required, opened a new log
I20260928 06:37:21.717054 4213 ts_tablet_manager.cc:1397] T 6dc698d297394664858732a519f2ed52 P 1ed91ab0a8e54c48ba26ae140d3d696d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 06:37:21.718277 4212 raft_consensus.cc:348] T 6dc698d297394664858732a519f2ed52 P f5bc49ea35e445dfb53e181c24e167f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ed91ab0a8e54c48ba26ae140d3d696d" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 46705 } } peers { permanent_uuid: "f5bc49ea35e445dfb53e181c24e167f1" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 42393 } } peers { permanent_uuid: "f0e81cf01a4a45678a109bfcf159bd30" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 36245 } }
I20260928 06:37:21.718448 4212 raft_consensus.cc:374] T 6dc698d297394664858732a519f2ed52 P f5bc49ea35e445dfb53e181c24e167f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:37:21.718493 4212 raft_consensus.cc:729] T 6dc698d297394664858732a519f2ed52 P f5bc49ea35e445dfb53e181c24e167f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5bc49ea35e445dfb53e181c24e167f1, State: Initialized, Role: FOLLOWER
I20260928 06:37:21.718621 4212 consensus_queue.cc:260] T 6dc698d297394664858732a519f2ed52 P f5bc49ea35e445dfb53e181c24e167f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ed91ab0a8e54c48ba26ae140d3d696d" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 46705 } } peers { permanent_uuid: "f5bc49ea35e445dfb53e181c24e167f1" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 42393 } } peers { permanent_uuid: "f0e81cf01a4a45678a109bfcf159bd30" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 36245 } }
I20260928 06:37:21.718918 4212 ts_tablet_manager.cc:1428] T 6dc698d297394664858732a519f2ed52 P f5bc49ea35e445dfb53e181c24e167f1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:37:21.718961 3927 heartbeater.cc:503] Master 127.16.157.126:39847 was elected leader, sending a full tablet report...
I20260928 06:37:21.719213 4213 raft_consensus.cc:348] T 6dc698d297394664858732a519f2ed52 P 1ed91ab0a8e54c48ba26ae140d3d696d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ed91ab0a8e54c48ba26ae140d3d696d" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 46705 } } peers { permanent_uuid: "f5bc49ea35e445dfb53e181c24e167f1" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 42393 } } peers { permanent_uuid: "f0e81cf01a4a45678a109bfcf159bd30" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 36245 } }
I20260928 06:37:21.719398 4213 raft_consensus.cc:374] T 6dc698d297394664858732a519f2ed52 P 1ed91ab0a8e54c48ba26ae140d3d696d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:37:21.719445 4213 raft_consensus.cc:729] T 6dc698d297394664858732a519f2ed52 P 1ed91ab0a8e54c48ba26ae140d3d696d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ed91ab0a8e54c48ba26ae140d3d696d, State: Initialized, Role: FOLLOWER
I20260928 06:37:21.719321 4214 raft_consensus.cc:348] T 6dc698d297394664858732a519f2ed52 P f0e81cf01a4a45678a109bfcf159bd30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ed91ab0a8e54c48ba26ae140d3d696d" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 46705 } } peers { permanent_uuid: "f5bc49ea35e445dfb53e181c24e167f1" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 42393 } } peers { permanent_uuid: "f0e81cf01a4a45678a109bfcf159bd30" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 36245 } }
I20260928 06:37:21.719509 4214 raft_consensus.cc:374] T 6dc698d297394664858732a519f2ed52 P f0e81cf01a4a45678a109bfcf159bd30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:37:21.719556 4214 raft_consensus.cc:729] T 6dc698d297394664858732a519f2ed52 P f0e81cf01a4a45678a109bfcf159bd30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e81cf01a4a45678a109bfcf159bd30, State: Initialized, Role: FOLLOWER
I20260928 06:37:21.719586 4213 consensus_queue.cc:260] T 6dc698d297394664858732a519f2ed52 P 1ed91ab0a8e54c48ba26ae140d3d696d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ed91ab0a8e54c48ba26ae140d3d696d" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 46705 } } peers { permanent_uuid: "f5bc49ea35e445dfb53e181c24e167f1" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 42393 } } peers { permanent_uuid: "f0e81cf01a4a45678a109bfcf159bd30" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 36245 } }
I20260928 06:37:21.719677 4214 consensus_queue.cc:260] T 6dc698d297394664858732a519f2ed52 P f0e81cf01a4a45678a109bfcf159bd30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ed91ab0a8e54c48ba26ae140d3d696d" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 46705 } } peers { permanent_uuid: "f5bc49ea35e445dfb53e181c24e167f1" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 42393 } } peers { permanent_uuid: "f0e81cf01a4a45678a109bfcf159bd30" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 36245 } }
I20260928 06:37:21.719877 4213 ts_tablet_manager.cc:1428] T 6dc698d297394664858732a519f2ed52 P 1ed91ab0a8e54c48ba26ae140d3d696d: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 06:37:21.720002 4061 heartbeater.cc:503] Master 127.16.157.126:39847 was elected leader, sending a full tablet report...
I20260928 06:37:21.720126 4195 heartbeater.cc:503] Master 127.16.157.126:39847 was elected leader, sending a full tablet report...
I20260928 06:37:21.720384 4214 ts_tablet_manager.cc:1428] T 6dc698d297394664858732a519f2ed52 P f0e81cf01a4a45678a109bfcf159bd30: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260928 06:37:21.772153 4218 raft_consensus.cc:482] T 6dc698d297394664858732a519f2ed52 P f5bc49ea35e445dfb53e181c24e167f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:37:21.772388 4218 raft_consensus.cc:504] T 6dc698d297394664858732a519f2ed52 P f5bc49ea35e445dfb53e181c24e167f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ed91ab0a8e54c48ba26ae140d3d696d" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 46705 } } peers { permanent_uuid: "f5bc49ea35e445dfb53e181c24e167f1" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 42393 } } peers { permanent_uuid: "f0e81cf01a4a45678a109bfcf159bd30" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 36245 } }
I20260928 06:37:21.772791 4218 leader_election.cc:288] T 6dc698d297394664858732a519f2ed52 P f5bc49ea35e445dfb53e181c24e167f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ed91ab0a8e54c48ba26ae140d3d696d (127.16.157.66:46705), f0e81cf01a4a45678a109bfcf159bd30 (127.16.157.67:36245)
W20260928 06:37:21.773080 4062 tablet.cc:2369] T 6dc698d297394664858732a519f2ed52 P 1ed91ab0a8e54c48ba26ae140d3d696d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:37:21.776312 4211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:21.773414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:40377 (local address 127.16.157.67:36245)
0928 06:37:21.773459 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:21.773461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:21.773524 (+ 63us) server_negotiation.cc:407] Connection header received
0928 06:37:21.773551 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:21.773553 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:21.773562 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:21.773619 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:37:21.773859 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:21.774360 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:21.775482 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:21.775946 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:21.776035 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:21.776043 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:21.776195 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 06:37:21.776239 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:37:21.776319 3929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:21.773018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.66:46705 (local address 127.16.157.65:60121)
0928 06:37:21.773089 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:21.773103 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:21.773163 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:21.773557 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:21.773559 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:21.773561 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:37:21.773676 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:21.773678 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:21.774348 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:21.774349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:21.775337 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:21.775339 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:21.776041 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:21.776042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:21.776129 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:21.776137 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 06:37:21.776210 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 06:37:21.776248 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":4}
I20260928 06:37:21.776327 4208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:21.773122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:60121 (local address 127.16.157.66:46705)
0928 06:37:21.773399 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:21.773402 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:21.773412 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:37:21.773443 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:21.773444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:21.773455 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:21.773515 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:37:21.773760 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:21.774271 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:21.775422 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:21.775946 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:21.776031 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:21.776041 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:21.776233 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 06:37:21.776272 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249}
I20260928 06:37:21.776324 4221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:21.773157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.67:36245 (local address 127.16.157.65:40377)
0928 06:37:21.773396 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:21.773408 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:21.773457 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:21.773646 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:21.773651 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:21.773653 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:37:21.773740 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:21.773741 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:21.774439 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:21.774440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:21.775365 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:21.775368 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:21.776039 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:21.776040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:21.776121 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:21.776125 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:37:21.776169 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 06:37:21.776229 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":126,"threads_started":1}
I20260928 06:37:21.776767 4149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dc698d297394664858732a519f2ed52" candidate_uuid: "f5bc49ea35e445dfb53e181c24e167f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e81cf01a4a45678a109bfcf159bd30" is_pre_election: true
I20260928 06:37:21.776783 4015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dc698d297394664858732a519f2ed52" candidate_uuid: "f5bc49ea35e445dfb53e181c24e167f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ed91ab0a8e54c48ba26ae140d3d696d" is_pre_election: true
I20260928 06:37:21.776889 4149 raft_consensus.cc:2454] T 6dc698d297394664858732a519f2ed52 P f0e81cf01a4a45678a109bfcf159bd30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5bc49ea35e445dfb53e181c24e167f1 in term 0.
I20260928 06:37:21.776923 4015 raft_consensus.cc:2454] T 6dc698d297394664858732a519f2ed52 P 1ed91ab0a8e54c48ba26ae140d3d696d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5bc49ea35e445dfb53e181c24e167f1 in term 0.
I20260928 06:37:21.777125 3814 leader_election.cc:302] T 6dc698d297394664858732a519f2ed52 P f5bc49ea35e445dfb53e181c24e167f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f0e81cf01a4a45678a109bfcf159bd30, f5bc49ea35e445dfb53e181c24e167f1; no voters:
I20260928 06:37:21.777295 4218 raft_consensus.cc:2781] T 6dc698d297394664858732a519f2ed52 P f5bc49ea35e445dfb53e181c24e167f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:37:21.777351 4218 raft_consensus.cc:482] T 6dc698d297394664858732a519f2ed52 P f5bc49ea35e445dfb53e181c24e167f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:37:21.777369 4218 raft_consensus.cc:3037] T 6dc698d297394664858732a519f2ed52 P f5bc49ea35e445dfb53e181c24e167f1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:37:21.778235 4218 raft_consensus.cc:504] T 6dc698d297394664858732a519f2ed52 P f5bc49ea35e445dfb53e181c24e167f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ed91ab0a8e54c48ba26ae140d3d696d" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 46705 } } peers { permanent_uuid: "f5bc49ea35e445dfb53e181c24e167f1" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 42393 } } peers { permanent_uuid: "f0e81cf01a4a45678a109bfcf159bd30" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 36245 } }
I20260928 06:37:21.778337 4218 leader_election.cc:288] T 6dc698d297394664858732a519f2ed52 P f5bc49ea35e445dfb53e181c24e167f1 [CANDIDATE]: Term 1 election: Requested vote from peers 1ed91ab0a8e54c48ba26ae140d3d696d (127.16.157.66:46705), f0e81cf01a4a45678a109bfcf159bd30 (127.16.157.67:36245)
I20260928 06:37:21.778508 4015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dc698d297394664858732a519f2ed52" candidate_uuid: "f5bc49ea35e445dfb53e181c24e167f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ed91ab0a8e54c48ba26ae140d3d696d"
I20260928 06:37:21.778543 4149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dc698d297394664858732a519f2ed52" candidate_uuid: "f5bc49ea35e445dfb53e181c24e167f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e81cf01a4a45678a109bfcf159bd30"
I20260928 06:37:21.778589 4015 raft_consensus.cc:3037] T 6dc698d297394664858732a519f2ed52 P 1ed91ab0a8e54c48ba26ae140d3d696d [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:37:21.778612 4149 raft_consensus.cc:3037] T 6dc698d297394664858732a519f2ed52 P f0e81cf01a4a45678a109bfcf159bd30 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:37:21.779497 4015 raft_consensus.cc:2454] T 6dc698d297394664858732a519f2ed52 P 1ed91ab0a8e54c48ba26ae140d3d696d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5bc49ea35e445dfb53e181c24e167f1 in term 1.
I20260928 06:37:21.779497 4149 raft_consensus.cc:2454] T 6dc698d297394664858732a519f2ed52 P f0e81cf01a4a45678a109bfcf159bd30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5bc49ea35e445dfb53e181c24e167f1 in term 1.
I20260928 06:37:21.779700 3814 leader_election.cc:302] T 6dc698d297394664858732a519f2ed52 P f5bc49ea35e445dfb53e181c24e167f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ed91ab0a8e54c48ba26ae140d3d696d, f5bc49ea35e445dfb53e181c24e167f1; no voters:
I20260928 06:37:21.779803 4218 raft_consensus.cc:2781] T 6dc698d297394664858732a519f2ed52 P f5bc49ea35e445dfb53e181c24e167f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:37:21.779950 4218 raft_consensus.cc:686] T 6dc698d297394664858732a519f2ed52 P f5bc49ea35e445dfb53e181c24e167f1 [term 1 LEADER]: Becoming Leader. State: Replica: f5bc49ea35e445dfb53e181c24e167f1, State: Running, Role: LEADER
I20260928 06:37:21.780045 4218 consensus_queue.cc:237] T 6dc698d297394664858732a519f2ed52 P f5bc49ea35e445dfb53e181c24e167f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ed91ab0a8e54c48ba26ae140d3d696d" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 46705 } } peers { permanent_uuid: "f5bc49ea35e445dfb53e181c24e167f1" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 42393 } } peers { permanent_uuid: "f0e81cf01a4a45678a109bfcf159bd30" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 36245 } }
I20260928 06:37:21.780880 3734 catalog_manager.cc:5495] T 6dc698d297394664858732a519f2ed52 P f5bc49ea35e445dfb53e181c24e167f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to f5bc49ea35e445dfb53e181c24e167f1 (127.16.157.65). New cstate: current_term: 1 leader_uuid: "f5bc49ea35e445dfb53e181c24e167f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ed91ab0a8e54c48ba26ae140d3d696d" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 46705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5bc49ea35e445dfb53e181c24e167f1" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 42393 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0e81cf01a4a45678a109bfcf159bd30" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 36245 } health_report { overall_health: UNKNOWN } } }
I20260928 06:37:21.800832 4210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:21.798423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42962 (local address 127.16.157.65:42393)
0928 06:37:21.798472 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:21.798474 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:21.798483 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:37:21.798524 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:21.798529 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:21.798549 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:21.798628 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:37:21.798782 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:21.799235 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:21.800011 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:21.800200 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:21.800275 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:21.800487 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:37:21.800582 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:37:21.800678 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:21.800704 (+ 26us) server_negotiation.cc:299] Negotiation successful
0928 06:37:21.800724 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:37:21.802816 4015 raft_consensus.cc:1261] T 6dc698d297394664858732a519f2ed52 P 1ed91ab0a8e54c48ba26ae140d3d696d [term 1 FOLLOWER]: Refusing update from remote peer f5bc49ea35e445dfb53e181c24e167f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:37:21.802816 4149 raft_consensus.cc:1261] T 6dc698d297394664858732a519f2ed52 P f0e81cf01a4a45678a109bfcf159bd30 [term 1 FOLLOWER]: Refusing update from remote peer f5bc49ea35e445dfb53e181c24e167f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:37:21.803192 4222 consensus_queue.cc:1035] T 6dc698d297394664858732a519f2ed52 P f5bc49ea35e445dfb53e181c24e167f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ed91ab0a8e54c48ba26ae140d3d696d" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 46705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:37:21.803359 4218 consensus_queue.cc:1035] T 6dc698d297394664858732a519f2ed52 P f5bc49ea35e445dfb53e181c24e167f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e81cf01a4a45678a109bfcf159bd30" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 36245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:37:21.805892 4226 mvcc.cc:205] Tried to move back new op lower bound from 7334205201621299200 to 7334205201531756544. Current Snapshot: MvccSnapshot[applied={T|T < 7334205201621299200}]
I20260928 06:37:21.807036 4227 mvcc.cc:205] Tried to move back new op lower bound from 7334205201621299200 to 7334205201531756544. Current Snapshot: MvccSnapshot[applied={T|T < 7334205201621299200 or (T in {7334205201621299200})}]
I20260928 06:37:21.809939 3734 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:37:21.810027 3734 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:37:21.810778 3734 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1ec7d17a8c534533a2aee104948074de: Sending DeleteTablet for 3 replicas of tablet 6dc698d297394664858732a519f2ed52
I20260928 06:37:21.811117 3861 tablet_service.cc:1514] Processing DeleteTablet for tablet 6dc698d297394664858732a519f2ed52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:37:21 UTC) from {username='kudu', principal='kudu/127.16.157.126@KRBTEST.COM'} at 127.0.0.1:42960
I20260928 06:37:21.811120 3995 tablet_service.cc:1514] Processing DeleteTablet for tablet 6dc698d297394664858732a519f2ed52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:37:21 UTC) from {username='kudu', principal='kudu/127.16.157.126@KRBTEST.COM'} at 127.0.0.1:36156
I20260928 06:37:21.811339 4235 tablet_replica.cc:330] T 6dc698d297394664858732a519f2ed52 P f5bc49ea35e445dfb53e181c24e167f1: stopping tablet replica
I20260928 06:37:21.811331 4129 tablet_service.cc:1514] Processing DeleteTablet for tablet 6dc698d297394664858732a519f2ed52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:37:21 UTC) from {username='kudu', principal='kudu/127.16.157.126@KRBTEST.COM'} at 127.0.0.1:53724
I20260928 06:37:21.811434 4235 raft_consensus.cc:2229] T 6dc698d297394664858732a519f2ed52 P f5bc49ea35e445dfb53e181c24e167f1 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:37:21.811512 4235 raft_consensus.cc:2258] T 6dc698d297394664858732a519f2ed52 P f5bc49ea35e445dfb53e181c24e167f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:37:21.811632 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 3798
I20260928 06:37:21.811781 4237 tablet_replica.cc:330] T 6dc698d297394664858732a519f2ed52 P f0e81cf01a4a45678a109bfcf159bd30: stopping tablet replica
I20260928 06:37:21.813153 4236 tablet_replica.cc:330] T 6dc698d297394664858732a519f2ed52 P 1ed91ab0a8e54c48ba26ae140d3d696d: stopping tablet replica
I20260928 06:37:21.813287 4236 raft_consensus.cc:2229] T 6dc698d297394664858732a519f2ed52 P 1ed91ab0a8e54c48ba26ae140d3d696d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:37:21.813323 4237 raft_consensus.cc:2229] T 6dc698d297394664858732a519f2ed52 P f0e81cf01a4a45678a109bfcf159bd30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:37:21.813395 4236 raft_consensus.cc:2258] T 6dc698d297394664858732a519f2ed52 P 1ed91ab0a8e54c48ba26ae140d3d696d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:37:21.813421 4237 raft_consensus.cc:2258] T 6dc698d297394664858732a519f2ed52 P f0e81cf01a4a45678a109bfcf159bd30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:37:21.813977 4237 ts_tablet_manager.cc:1905] T 6dc698d297394664858732a519f2ed52 P f0e81cf01a4a45678a109bfcf159bd30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:37:21.814006 4236 ts_tablet_manager.cc:1905] T 6dc698d297394664858732a519f2ed52 P 1ed91ab0a8e54c48ba26ae140d3d696d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:37:21.815470 4237 ts_tablet_manager.cc:1918] T 6dc698d297394664858732a519f2ed52 P f0e81cf01a4a45678a109bfcf159bd30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:37:21.815558 4237 log.cc:1196] T 6dc698d297394664858732a519f2ed52 P f0e81cf01a4a45678a109bfcf159bd30: Deleting WAL directory at /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/minicluster-data/ts-2/wal/wals/6dc698d297394664858732a519f2ed52
I20260928 06:37:21.815819 4237 ts_tablet_manager.cc:1939] T 6dc698d297394664858732a519f2ed52 P f0e81cf01a4a45678a109bfcf159bd30: Deleting consensus metadata
I20260928 06:37:21.815965 4236 ts_tablet_manager.cc:1918] T 6dc698d297394664858732a519f2ed52 P 1ed91ab0a8e54c48ba26ae140d3d696d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:37:21.816042 4236 log.cc:1196] T 6dc698d297394664858732a519f2ed52 P 1ed91ab0a8e54c48ba26ae140d3d696d: Deleting WAL directory at /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577196946045-17013-0/minicluster-data/ts-1/wal/wals/6dc698d297394664858732a519f2ed52
I20260928 06:37:21.816260 3720 catalog_manager.cc:4841] TS f0e81cf01a4a45678a109bfcf159bd30 (127.16.157.67:36245): tablet 6dc698d297394664858732a519f2ed52 (table test-table [id=ef1ff00ce2974f0cb851f0785d56a734]) successfully deleted
I20260928 06:37:21.816411 4236 ts_tablet_manager.cc:1939] T 6dc698d297394664858732a519f2ed52 P 1ed91ab0a8e54c48ba26ae140d3d696d: Deleting consensus metadata
I20260928 06:37:21.816819 3720 catalog_manager.cc:4841] TS 1ed91ab0a8e54c48ba26ae140d3d696d (127.16.157.66:46705): tablet 6dc698d297394664858732a519f2ed52 (table test-table [id=ef1ff00ce2974f0cb851f0785d56a734]) successfully deleted
W20260928 06:37:21.818444 3722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.157.65:42393 (error 108)
I20260928 06:37:21.818657 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 3932
I20260928 06:37:21.818946 4207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:21.818828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.65:42393 (local address 127.0.0.1:42978)
0928 06:37:21.818875 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:21.818909 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.65:42393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 06:37:21.819042 3722 catalog_manager.cc:4584] TS f5bc49ea35e445dfb53e181c24e167f1 (127.16.157.65:42393): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6dc698d297394664858732a519f2ed52: Network error: Client connection negotiation failed: client connection to 127.16.157.65:42393: connect: Connection refused (error 111)
I20260928 06:37:21.824400 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 4066
I20260928 06:37:21.829211 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 3704
2026-09-28T06:37:21Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1976 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T06:37:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:37:21Z Disabled control of system clock
I20260928 06:37:21.856659 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577196946045-17013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577196946045-17013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577196946045-17013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:35183
--webserver_interface=127.16.157.126
--webserver_port=0
--builtin_ntp_servers=127.16.157.84:45653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.157.126:35183
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 06:37:21.936167 4242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:21.936381 4242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:21.936401 4242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:21.937547 4242 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:37:21.937615 4242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:21.937628 4242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:21.937641 4242 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:37:21.937697 4242 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:37:21.938997 4242 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:45653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577196946045-17013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577196946045-17013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.157.126:35183
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:35183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577196946045-17013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:37:21.939254 4242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:21.939540 4242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:21.944802 4250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:21.944826 4247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:21.944815 4248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:21.944869 4242 server_base.cc:1034] running on GCE node
I20260928 06:37:21.945253 4242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:21.945497 4242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:21.946702 4242 hybrid_clock.cc:648] HybridClock initialized: now 1790577441946685 us; error 34 us; skew 500 ppm
I20260928 06:37:21.947876 4242 webserver.cc:460] Webserver started at http://127.16.157.126:42091/ using document root <none> and password file <none>
I20260928 06:37:21.948130 4242 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:21.948217 4242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:21.948378 4242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:37:21.949476 4242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577196946045-17013-0/minicluster-data/master-0/data/instance:
uuid: "6843ec1355dd404787658d74609e90df"
format_stamp: "Formatted at 2026-09-28 06:37:21 on dist-test-slave-rrj5"
server_key: "280c13f3df98e8a79bf9e6a7bbbc3d19"
server_key_iv: "7dc1193ac6dd1570e2b2cb76baaecef0"
server_key_version: "encryptionkey@0"
I20260928 06:37:21.949931 4242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577196946045-17013-0/minicluster-data/master-0/wal/instance:
uuid: "6843ec1355dd404787658d74609e90df"
format_stamp: "Formatted at 2026-09-28 06:37:21 on dist-test-slave-rrj5"
server_key: "280c13f3df98e8a79bf9e6a7bbbc3d19"
server_key_iv: "7dc1193ac6dd1570e2b2cb76baaecef0"
server_key_version: "encryptionkey@0"
I20260928 06:37:21.951241 4242 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:37:21.951927 4256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:21.952086 4242 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:37:21.952193 4242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577196946045-17013-0/minicluster-data/master-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577196946045-17013-0/minicluster-data/master-0/wal
uuid: "6843ec1355dd404787658d74609e90df"
format_stamp: "Formatted at 2026-09-28 06:37:21 on dist-test-slave-rrj5"
server_key: "280c13f3df98e8a79bf9e6a7bbbc3d19"
server_key_iv: "7dc1193ac6dd1570e2b2cb76baaecef0"
server_key_version: "encryptionkey@0"
I20260928 06:37:21.952276 4242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577196946045-17013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577196946045-17013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577196946045-17013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:21.974043 4242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:21.974367 4242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:21.974525 4242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:21.978384 4242 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.126:35183
I20260928 06:37:21.978425 4308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.126:35183 every 8 connection(s)
I20260928 06:37:21.978791 4242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
I20260928 06:37:21.979770 4309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:37:21.981371 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 4242
I20260928 06:37:21.981494 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577196946045-17013-0/minicluster-data/master-0/wal/instance
I20260928 06:37:21.981719 17013 external_mini_cluster.cc:1428] Setting key 022639d9f5b2c28db1d3cc8d91961733
I20260928 06:37:21.982340 4309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6843ec1355dd404787658d74609e90df: Bootstrap starting.
I20260928 06:37:21.982959 4309 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6843ec1355dd404787658d74609e90df: Neither blocks nor log segments found. Creating new log.
I20260928 06:37:21.983261 4309 log.cc:824] T 00000000000000000000000000000000 P 6843ec1355dd404787658d74609e90df: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:21.983987 4309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6843ec1355dd404787658d74609e90df: No bootstrap required, opened a new log
I20260928 06:37:21.986155 4309 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6843ec1355dd404787658d74609e90df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6843ec1355dd404787658d74609e90df" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 35183 } }
I20260928 06:37:21.986325 4309 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6843ec1355dd404787658d74609e90df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:37:21.986367 4309 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6843ec1355dd404787658d74609e90df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6843ec1355dd404787658d74609e90df, State: Initialized, Role: FOLLOWER
I20260928 06:37:21.986526 4309 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6843ec1355dd404787658d74609e90df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6843ec1355dd404787658d74609e90df" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 35183 } }
I20260928 06:37:21.986630 4309 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6843ec1355dd404787658d74609e90df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:37:21.986687 4309 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6843ec1355dd404787658d74609e90df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:37:21.986744 4309 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6843ec1355dd404787658d74609e90df [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:37:21.987396 4309 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6843ec1355dd404787658d74609e90df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6843ec1355dd404787658d74609e90df" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 35183 } }
I20260928 06:37:21.987519 4309 leader_election.cc:302] T 00000000000000000000000000000000 P 6843ec1355dd404787658d74609e90df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6843ec1355dd404787658d74609e90df; no voters:
I20260928 06:37:21.987694 4309 leader_election.cc:288] T 00000000000000000000000000000000 P 6843ec1355dd404787658d74609e90df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:37:21.987835 4314 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6843ec1355dd404787658d74609e90df [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:37:21.987994 4312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:21.982298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56522 (local address 127.16.157.126:35183)
0928 06:37:21.982491 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:21.982498 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:21.982523 (+ 25us) server_negotiation.cc:407] Connection header received
0928 06:37:21.982921 (+ 398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:21.982927 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:21.982985 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:21.983138 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:21.983481 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:21.984823 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:21.985609 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:21.986279 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:21.986406 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:21.987210 (+ 804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:21.987214 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:21.987214 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:37:21.987372 (+ 158us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:21.987403 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:37:21.987413 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:21.987464 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:21.987644 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 06:37:21.987730 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":86,"threads_started":1}
I20260928 06:37:21.987979 4309 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6843ec1355dd404787658d74609e90df [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:37:21.988056 4314 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6843ec1355dd404787658d74609e90df [term 1 LEADER]: Becoming Leader. State: Replica: 6843ec1355dd404787658d74609e90df, State: Running, Role: LEADER
I20260928 06:37:21.988350 4314 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6843ec1355dd404787658d74609e90df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6843ec1355dd404787658d74609e90df" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 35183 } }
I20260928 06:37:21.988842 4316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6843ec1355dd404787658d74609e90df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6843ec1355dd404787658d74609e90df. Latest consensus state: current_term: 1 leader_uuid: "6843ec1355dd404787658d74609e90df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6843ec1355dd404787658d74609e90df" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 35183 } } }
I20260928 06:37:21.989004 4316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6843ec1355dd404787658d74609e90df [sys.catalog]: This master's current role is: LEADER
I20260928 06:37:21.989148 4315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6843ec1355dd404787658d74609e90df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6843ec1355dd404787658d74609e90df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6843ec1355dd404787658d74609e90df" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 35183 } } }
I20260928 06:37:21.989319 4315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6843ec1355dd404787658d74609e90df [sys.catalog]: This master's current role is: LEADER
I20260928 06:37:21.989538 4329 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:37:21.990188 4329 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:37:21.991628 4329 catalog_manager.cc:1347] Generated new cluster ID: 2ccd7cfcf407429ab2db9a83d993139f
I20260928 06:37:21.991701 4329 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:37:22.003805 4329 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:37:22.004333 4329 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:37:22.010478 4329 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6843ec1355dd404787658d74609e90df: Generated new TSK 0
I20260928 06:37:22.010656 4329 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:37:22.015621 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577196946045-17013-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577196946045-17013-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577196946045-17013-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.65:0
--local_ip_for_outbound_sockets=127.16.157.65
--webserver_interface=127.16.157.65
--webserver_port=0
--tserver_master_addrs=127.16.157.126:35183
--builtin_ntp_servers=127.16.157.84:45653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 06:37:22.105530 4333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:22.105811 4333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:22.105834 4333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:22.107013 4333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:22.107059 4333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:22.107085 4333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.65
I20260928 06:37:22.108520 4333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:45653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577196946045-17013-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577196946045-17013-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.157.65
--webserver_port=0
--tserver_master_addrs=127.16.157.126:35183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.65
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577196946045-17013-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:37:22.108772 4333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:22.109122 4333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:22.114322 4338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:22.114369 4341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:22.114413 4339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:22.114795 4333 server_base.cc:1034] running on GCE node
I20260928 06:37:22.115005 4333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:22.115253 4333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:22.116423 4333 hybrid_clock.cc:648] HybridClock initialized: now 1790577442116408 us; error 29 us; skew 500 ppm
I20260928 06:37:22.117834 4333 webserver.cc:460] Webserver started at http://127.16.157.65:36567/ using document root <none> and password file <none>
I20260928 06:37:22.118099 4333 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:22.118170 4333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:22.118332 4333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:37:22.119426 4333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577196946045-17013-0/minicluster-data/ts-0/data/instance:
uuid: "0def6aa011f14329ad489b2311ac46a1"
format_stamp: "Formatted at 2026-09-28 06:37:22 on dist-test-slave-rrj5"
server_key: "068134f719fb7c5d9268b879bf9af0b7"
server_key_iv: "fe4e2a47e195b6bf8d10c31006ffae49"
server_key_version: "encryptionkey@0"
I20260928 06:37:22.119800 4333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577196946045-17013-0/minicluster-data/ts-0/wal/instance:
uuid: "0def6aa011f14329ad489b2311ac46a1"
format_stamp: "Formatted at 2026-09-28 06:37:22 on dist-test-slave-rrj5"
server_key: "068134f719fb7c5d9268b879bf9af0b7"
server_key_iv: "fe4e2a47e195b6bf8d10c31006ffae49"
server_key_version: "encryptionkey@0"
I20260928 06:37:22.121212 4333 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:37:22.122068 4347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:22.122231 4333 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:22.122336 4333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577196946045-17013-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577196946045-17013-0/minicluster-data/ts-0/wal
uuid: "0def6aa011f14329ad489b2311ac46a1"
format_stamp: "Formatted at 2026-09-28 06:37:22 on dist-test-slave-rrj5"
server_key: "068134f719fb7c5d9268b879bf9af0b7"
server_key_iv: "fe4e2a47e195b6bf8d10c31006ffae49"
server_key_version: "encryptionkey@0"
I20260928 06:37:22.122442 4333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577196946045-17013-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577196946045-17013-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577196946045-17013-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:22.133368 4333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:22.133795 4333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:22.133962 4333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:22.134210 4333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:37:22.134555 4333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:37:22.134613 4333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:22.134660 4333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:37:22.134698 4333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:22.140877 4333 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.65:39991
I20260928 06:37:22.140918 4460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.65:39991 every 8 connection(s)
I20260928 06:37:22.141325 4333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577196946045-17013-0/minicluster-data/ts-0/data/info.pb
I20260928 06:37:22.146646 4312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:22.142183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.65:55117 (local address 127.16.157.126:35183)
0928 06:37:22.142258 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:22.142260 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:22.142499 (+ 239us) server_negotiation.cc:407] Connection header received
0928 06:37:22.142998 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:22.142999 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:22.143011 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:22.143078 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:22.143761 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:22.144262 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:22.145258 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:22.145452 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:22.145508 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:22.146414 (+ 906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:22.146416 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:22.146416 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:37:22.146420 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:22.146434 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:37:22.146438 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:22.146482 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:22.146561 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 06:37:22.146585 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 06:37:22.146852 4463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:22.142143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:35183 (local address 127.16.157.65:55117)
0928 06:37:22.142356 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:22.142397 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:22.142917 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:22.143154 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:22.143157 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:22.143166 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:37:22.143688 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:22.143694 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:22.144349 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:22.144350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:22.145168 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:22.145170 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:22.145552 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:22.145553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:22.145756 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:22.146266 (+ 510us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:37:22.146266 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:37:22.146296 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:37:22.146324 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:37:22.146492 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:37:22.146496 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:37:22.146499 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:37:22.146550 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 06:37:22.146631 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":112,"threads_started":1}
I20260928 06:37:22.147441 4461 heartbeater.cc:347] Connected to a master server at 127.16.157.126:35183
I20260928 06:37:22.147549 4461 heartbeater.cc:464] Registering TS with master...
I20260928 06:37:22.147739 4461 heartbeater.cc:511] Master 127.16.157.126:35183 requested a full tablet report, sending...
I20260928 06:37:22.148263 4273 ts_manager.cc:194] Registered new tserver with Master: 0def6aa011f14329ad489b2311ac46a1 (127.16.157.65:39991)
I20260928 06:37:22.148840 4273 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.157.65:55117
I20260928 06:37:22.150978 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 4333
I20260928 06:37:22.151090 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577196946045-17013-0/minicluster-data/ts-0/wal/instance
I20260928 06:37:22.151296 17013 external_mini_cluster.cc:1428] Setting key 2cab1edd33d15677b842925395b0da9d
I20260928 06:37:22.152220 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577196946045-17013-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577196946045-17013-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577196946045-17013-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.66:0
--local_ip_for_outbound_sockets=127.16.157.66
--webserver_interface=127.16.157.66
--webserver_port=0
--tserver_master_addrs=127.16.157.126:35183
--builtin_ntp_servers=127.16.157.84:45653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 06:37:22.234282 4464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:22.234505 4464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:22.234529 4464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:22.235666 4464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:22.235718 4464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:22.235762 4464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.66
I20260928 06:37:22.237042 4464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:45653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577196946045-17013-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577196946045-17013-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.157.66
--webserver_port=0
--tserver_master_addrs=127.16.157.126:35183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.66
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577196946045-17013-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:37:22.237244 4464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:22.237498 4464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:22.242683 4470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:22.242719 4469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:22.242748 4464 server_base.cc:1034] running on GCE node
W20260928 06:37:22.242666 4472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:22.243170 4464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:22.243453 4464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:22.244628 4464 hybrid_clock.cc:648] HybridClock initialized: now 1790577442244614 us; error 27 us; skew 500 ppm
I20260928 06:37:22.245993 4464 webserver.cc:460] Webserver started at http://127.16.157.66:42553/ using document root <none> and password file <none>
I20260928 06:37:22.246248 4464 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:22.246318 4464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:22.246466 4464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:37:22.247581 4464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577196946045-17013-0/minicluster-data/ts-1/data/instance:
uuid: "c8627e98524e4d09ba07b79634e1033f"
format_stamp: "Formatted at 2026-09-28 06:37:22 on dist-test-slave-rrj5"
server_key: "92d598eae54d09511dce302f6f009b73"
server_key_iv: "0b2d5dda3ab80056e06f23bcd1033d12"
server_key_version: "encryptionkey@0"
I20260928 06:37:22.247946 4464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577196946045-17013-0/minicluster-data/ts-1/wal/instance:
uuid: "c8627e98524e4d09ba07b79634e1033f"
format_stamp: "Formatted at 2026-09-28 06:37:22 on dist-test-slave-rrj5"
server_key: "92d598eae54d09511dce302f6f009b73"
server_key_iv: "0b2d5dda3ab80056e06f23bcd1033d12"
server_key_version: "encryptionkey@0"
I20260928 06:37:22.249243 4464 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:37:22.250066 4478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:22.250247 4464 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:37:22.250365 4464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577196946045-17013-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577196946045-17013-0/minicluster-data/ts-1/wal
uuid: "c8627e98524e4d09ba07b79634e1033f"
format_stamp: "Formatted at 2026-09-28 06:37:22 on dist-test-slave-rrj5"
server_key: "92d598eae54d09511dce302f6f009b73"
server_key_iv: "0b2d5dda3ab80056e06f23bcd1033d12"
server_key_version: "encryptionkey@0"
I20260928 06:37:22.250440 4464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577196946045-17013-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577196946045-17013-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577196946045-17013-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:22.275538 4464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:22.275954 4464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:22.276111 4464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:22.276372 4464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:37:22.276751 4464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:37:22.276808 4464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:22.276876 4464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:37:22.276901 4464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:22.282934 4464 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.66:45109
I20260928 06:37:22.282986 4591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.66:45109 every 8 connection(s)
I20260928 06:37:22.283363 4464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577196946045-17013-0/minicluster-data/ts-1/data/info.pb
I20260928 06:37:22.287899 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 4464
I20260928 06:37:22.288029 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577196946045-17013-0/minicluster-data/ts-1/wal/instance
I20260928 06:37:22.288241 17013 external_mini_cluster.cc:1428] Setting key b8ffb2c0cf67237b37e41a05452ab159
I20260928 06:37:22.288908 4312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:22.284150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.66:48713 (local address 127.16.157.126:35183)
0928 06:37:22.284236 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:22.284239 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:22.284465 (+ 226us) server_negotiation.cc:407] Connection header received
0928 06:37:22.285007 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:22.285008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:22.285018 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:22.285073 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:22.285842 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:22.286349 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:22.287330 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:22.287535 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:22.287594 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:22.288567 (+ 973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:22.288569 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:22.288569 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:37:22.288575 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:22.288595 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:37:22.288599 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:22.288657 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:22.288797 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 06:37:22.288828 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 06:37:22.289143 4594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:22.284145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:35183 (local address 127.16.157.66:48713)
0928 06:37:22.284337 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:22.284373 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:22.284893 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:22.285128 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:22.285131 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:22.285142 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:37:22.285760 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:22.285766 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:22.286440 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:22.286440 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:22.287253 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:22.287256 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:22.287618 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:22.287619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:22.287749 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:22.288382 (+ 633us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:37:22.288382 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:37:22.288430 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:37:22.288463 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:37:22.288685 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:37:22.288693 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:37:22.288697 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:37:22.288788 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 06:37:22.288923 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":102,"threads_started":1}
I20260928 06:37:22.289379 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577196946045-17013-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577196946045-17013-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577196946045-17013-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.157.67:0
--local_ip_for_outbound_sockets=127.16.157.67
--webserver_interface=127.16.157.67
--webserver_port=0
--tserver_master_addrs=127.16.157.126:35183
--builtin_ntp_servers=127.16.157.84:45653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 06:37:22.289644 4592 heartbeater.cc:347] Connected to a master server at 127.16.157.126:35183
I20260928 06:37:22.289738 4592 heartbeater.cc:464] Registering TS with master...
I20260928 06:37:22.289944 4592 heartbeater.cc:511] Master 127.16.157.126:35183 requested a full tablet report, sending...
I20260928 06:37:22.290380 4273 ts_manager.cc:194] Registered new tserver with Master: c8627e98524e4d09ba07b79634e1033f (127.16.157.66:45109)
I20260928 06:37:22.290745 4273 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.157.66:48713
W20260928 06:37:22.373159 4595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:22.373371 4595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:22.373392 4595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:22.374691 4595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:22.374739 4595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:22.374765 4595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.157.67
I20260928 06:37:22.376080 4595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:45653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577196946045-17013-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577196946045-17013-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.157.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.157.67
--webserver_port=0
--tserver_master_addrs=127.16.157.126:35183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.157.67
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577196946045-17013-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:37:22.376286 4595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:22.376535 4595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:22.381762 4601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:22.381829 4600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:22.381800 4603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:22.382062 4595 server_base.cc:1034] running on GCE node
I20260928 06:37:22.382303 4595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:22.382519 4595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:22.383699 4595 hybrid_clock.cc:648] HybridClock initialized: now 1790577442383689 us; error 41 us; skew 500 ppm
I20260928 06:37:22.385097 4595 webserver.cc:460] Webserver started at http://127.16.157.67:41585/ using document root <none> and password file <none>
I20260928 06:37:22.385371 4595 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:22.385442 4595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:22.385627 4595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:37:22.386736 4595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577196946045-17013-0/minicluster-data/ts-2/data/instance:
uuid: "8e96ff6bffdb4b0983159e3b967752f5"
format_stamp: "Formatted at 2026-09-28 06:37:22 on dist-test-slave-rrj5"
server_key: "c9c1407f7688c09cd816fd5d225acb79"
server_key_iv: "917323cbc82af019c6bc7eb31f4cf5c6"
server_key_version: "encryptionkey@0"
I20260928 06:37:22.387109 4595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577196946045-17013-0/minicluster-data/ts-2/wal/instance:
uuid: "8e96ff6bffdb4b0983159e3b967752f5"
format_stamp: "Formatted at 2026-09-28 06:37:22 on dist-test-slave-rrj5"
server_key: "c9c1407f7688c09cd816fd5d225acb79"
server_key_iv: "917323cbc82af019c6bc7eb31f4cf5c6"
server_key_version: "encryptionkey@0"
I20260928 06:37:22.388536 4595 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:37:22.389423 4609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:22.389626 4595 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:37:22.389730 4595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577196946045-17013-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577196946045-17013-0/minicluster-data/ts-2/wal
uuid: "8e96ff6bffdb4b0983159e3b967752f5"
format_stamp: "Formatted at 2026-09-28 06:37:22 on dist-test-slave-rrj5"
server_key: "c9c1407f7688c09cd816fd5d225acb79"
server_key_iv: "917323cbc82af019c6bc7eb31f4cf5c6"
server_key_version: "encryptionkey@0"
I20260928 06:37:22.389814 4595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577196946045-17013-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577196946045-17013-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577196946045-17013-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:22.406718 4595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:22.407069 4595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:22.407230 4595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:22.407486 4595 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:37:22.407850 4595 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:37:22.407905 4595 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:22.407948 4595 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:37:22.407979 4595 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:22.414238 4595 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.67:44713
I20260928 06:37:22.414328 4722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.67:44713 every 8 connection(s)
I20260928 06:37:22.414721 4595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577196946045-17013-0/minicluster-data/ts-2/data/info.pb
I20260928 06:37:22.420177 4312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:22.415498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.67:56905 (local address 127.16.157.126:35183)
0928 06:37:22.415547 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:22.415548 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:22.415799 (+ 251us) server_negotiation.cc:407] Connection header received
0928 06:37:22.416382 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:22.416384 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:22.416397 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:22.416449 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:22.417161 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:22.417749 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:22.418758 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:22.418967 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:22.419045 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:22.419855 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:22.419856 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:22.419856 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:37:22.419861 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:22.419874 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:37:22.419878 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:22.419954 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:22.420052 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 06:37:22.420079 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 06:37:22.420439 4725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:22.415468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:35183 (local address 127.16.157.67:56905)
0928 06:37:22.415674 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:22.415707 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:22.416253 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:22.416527 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:22.416531 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:22.416538 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:37:22.417043 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:22.417048 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:22.417848 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:22.417849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:22.418664 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:22.418666 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:22.419054 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:22.419056 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:22.419194 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:22.419703 (+ 509us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:37:22.419703 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:37:22.419731 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:37:22.419754 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:37:22.419956 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:37:22.419962 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:37:22.419966 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:37:22.420052 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 06:37:22.420187 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":88,"threads_started":1}
I20260928 06:37:22.420948 4723 heartbeater.cc:347] Connected to a master server at 127.16.157.126:35183
I20260928 06:37:22.421026 4723 heartbeater.cc:464] Registering TS with master...
I20260928 06:37:22.421209 4723 heartbeater.cc:511] Master 127.16.157.126:35183 requested a full tablet report, sending...
I20260928 06:37:22.421630 4273 ts_manager.cc:194] Registered new tserver with Master: 8e96ff6bffdb4b0983159e3b967752f5 (127.16.157.67:44713)
I20260928 06:37:22.422006 4273 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.157.67:56905
I20260928 06:37:22.424831 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 4595
I20260928 06:37:22.424924 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577196946045-17013-0/minicluster-data/ts-2/wal/instance
I20260928 06:37:22.425077 17013 external_mini_cluster.cc:1428] Setting key e3eb6a555ca2eab6f23cd7770870e153
I20260928 06:37:22.426033 17013 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:37:22.430008 4312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:22.427410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56526 (local address 127.16.157.126:35183)
0928 06:37:22.427466 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:22.427468 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:22.427582 (+ 114us) server_negotiation.cc:407] Connection header received
0928 06:37:22.427604 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:22.427605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:22.427613 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:22.427663 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:22.427812 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:22.428254 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:22.428940 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:22.429111 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:22.429161 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:22.429839 (+ 678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:22.429840 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:22.429841 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:37:22.429845 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:22.429856 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:37:22.429860 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:22.429901 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:22.429934 (+ 33us) server_negotiation.cc:299] Negotiation successful
0928 06:37:22.429953 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 06:37:22.431499 4273 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56526:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:37:22.432996 4273 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:37:22.439543 4735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:22.435437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.65:39991 (local address 127.0.0.1:46362)
0928 06:37:22.435653 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:22.435662 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:22.435704 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:22.436103 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:22.436103 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:22.436105 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:37:22.436148 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:22.436149 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:22.437041 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:22.437042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:22.438255 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:22.438258 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:22.439229 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:22.439230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:22.439312 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:22.439328 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 06:37:22.439394 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 06:37:22.439448 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":1,"thread_start_us":68,"threads_started":1}
I20260928 06:37:22.439769 4732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:22.435215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.67:44713 (local address 127.0.0.1:38752)
0928 06:37:22.435524 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:22.435569 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:22.435683 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:22.436030 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:22.436033 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:22.436035 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:37:22.436110 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:22.436112 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:22.437460 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:22.437461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:22.438488 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:22.438490 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:22.439370 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:22.439371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:22.439437 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:22.439441 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:37:22.439497 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 06:37:22.439541 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":78,"threads_started":1}
I20260928 06:37:22.439819 4733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:22.435333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38752 (local address 127.16.157.67:44713)
0928 06:37:22.435492 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:22.435498 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:22.435780 (+ 282us) server_negotiation.cc:407] Connection header received
0928 06:37:22.435844 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:22.435847 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:22.435865 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:22.435942 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:37:22.436182 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:22.437361 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:22.438581 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:22.439299 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:22.439367 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:22.439455 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:22.439540 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 06:37:22.439584 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":81,"threads_started":1}
I20260928 06:37:22.439899 4736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:22.435665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46362 (local address 127.16.157.65:39991)
0928 06:37:22.435901 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:22.435906 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:22.435914 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:37:22.435953 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:22.435955 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:22.435968 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:22.436039 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:37:22.436218 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:22.436938 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:22.438361 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:22.439100 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:22.439175 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:22.439292 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:22.439764 (+ 472us) server_negotiation.cc:299] Negotiation successful
0928 06:37:22.439812 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":96,"threads_started":1}
I20260928 06:37:22.440642 4731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:22.435257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.66:45109 (local address 127.0.0.1:35494)
0928 06:37:22.435600 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:22.435611 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:22.435682 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:22.436047 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:22.436048 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:22.436050 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:37:22.436121 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:22.436124 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:22.438556 (+ 2432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:22.438557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:22.439430 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:22.439432 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:22.440421 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:22.440422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:22.440483 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:22.440487 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:37:22.440531 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 06:37:22.440573 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":109,"threads_started":1}
I20260928 06:37:22.440658 4734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:22.435389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35494 (local address 127.16.157.66:45109)
0928 06:37:22.435829 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:22.435833 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:22.435842 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:37:22.435882 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:22.435884 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:22.435900 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:22.435974 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:37:22.436221 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:22.437375 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:22.439520 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:22.440315 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:22.440417 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:22.440512 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:22.440542 (+ 30us) server_negotiation.cc:299] Negotiation successful
0928 06:37:22.440579 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":107,"threads_started":1}
I20260928 06:37:22.440608 4395 tablet_service.cc:1467] Processing CreateTablet for tablet 5b9518f1d0f144af8bead119dcf7770a (DEFAULT_TABLE table=test-table [id=9444b4d687a1446a9a42c5e6bbc24578]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:37:22.440608 4657 tablet_service.cc:1467] Processing CreateTablet for tablet 5b9518f1d0f144af8bead119dcf7770a (DEFAULT_TABLE table=test-table [id=9444b4d687a1446a9a42c5e6bbc24578]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:37:22.440986 4657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b9518f1d0f144af8bead119dcf7770a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:37:22.440986 4395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b9518f1d0f144af8bead119dcf7770a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:37:22.441268 4526 tablet_service.cc:1467] Processing CreateTablet for tablet 5b9518f1d0f144af8bead119dcf7770a (DEFAULT_TABLE table=test-table [id=9444b4d687a1446a9a42c5e6bbc24578]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:37:22.441521 4526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b9518f1d0f144af8bead119dcf7770a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:37:22.444027 4737 tablet_bootstrap.cc:492] T 5b9518f1d0f144af8bead119dcf7770a P c8627e98524e4d09ba07b79634e1033f: Bootstrap starting.
I20260928 06:37:22.444602 4739 tablet_bootstrap.cc:492] T 5b9518f1d0f144af8bead119dcf7770a P 0def6aa011f14329ad489b2311ac46a1: Bootstrap starting.
I20260928 06:37:22.444770 4737 tablet_bootstrap.cc:654] T 5b9518f1d0f144af8bead119dcf7770a P c8627e98524e4d09ba07b79634e1033f: Neither blocks nor log segments found. Creating new log.
I20260928 06:37:22.444955 4738 tablet_bootstrap.cc:492] T 5b9518f1d0f144af8bead119dcf7770a P 8e96ff6bffdb4b0983159e3b967752f5: Bootstrap starting.
I20260928 06:37:22.445143 4737 log.cc:824] T 5b9518f1d0f144af8bead119dcf7770a P c8627e98524e4d09ba07b79634e1033f: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:22.445477 4739 tablet_bootstrap.cc:654] T 5b9518f1d0f144af8bead119dcf7770a P 0def6aa011f14329ad489b2311ac46a1: Neither blocks nor log segments found. Creating new log.
I20260928 06:37:22.445724 4738 tablet_bootstrap.cc:654] T 5b9518f1d0f144af8bead119dcf7770a P 8e96ff6bffdb4b0983159e3b967752f5: Neither blocks nor log segments found. Creating new log.
I20260928 06:37:22.445859 4739 log.cc:824] T 5b9518f1d0f144af8bead119dcf7770a P 0def6aa011f14329ad489b2311ac46a1: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:22.446046 4738 log.cc:824] T 5b9518f1d0f144af8bead119dcf7770a P 8e96ff6bffdb4b0983159e3b967752f5: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:22.446110 4737 tablet_bootstrap.cc:492] T 5b9518f1d0f144af8bead119dcf7770a P c8627e98524e4d09ba07b79634e1033f: No bootstrap required, opened a new log
I20260928 06:37:22.446189 4737 ts_tablet_manager.cc:1397] T 5b9518f1d0f144af8bead119dcf7770a P c8627e98524e4d09ba07b79634e1033f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:37:22.446750 4739 tablet_bootstrap.cc:492] T 5b9518f1d0f144af8bead119dcf7770a P 0def6aa011f14329ad489b2311ac46a1: No bootstrap required, opened a new log
I20260928 06:37:22.446810 4739 ts_tablet_manager.cc:1397] T 5b9518f1d0f144af8bead119dcf7770a P 0def6aa011f14329ad489b2311ac46a1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:37:22.448462 4737 raft_consensus.cc:348] T 5b9518f1d0f144af8bead119dcf7770a P c8627e98524e4d09ba07b79634e1033f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e96ff6bffdb4b0983159e3b967752f5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 44713 } } peers { permanent_uuid: "c8627e98524e4d09ba07b79634e1033f" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 45109 } } peers { permanent_uuid: "0def6aa011f14329ad489b2311ac46a1" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 39991 } }
I20260928 06:37:22.448621 4737 raft_consensus.cc:374] T 5b9518f1d0f144af8bead119dcf7770a P c8627e98524e4d09ba07b79634e1033f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:37:22.448675 4737 raft_consensus.cc:729] T 5b9518f1d0f144af8bead119dcf7770a P c8627e98524e4d09ba07b79634e1033f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8627e98524e4d09ba07b79634e1033f, State: Initialized, Role: FOLLOWER
I20260928 06:37:22.448798 4737 consensus_queue.cc:260] T 5b9518f1d0f144af8bead119dcf7770a P c8627e98524e4d09ba07b79634e1033f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e96ff6bffdb4b0983159e3b967752f5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 44713 } } peers { permanent_uuid: "c8627e98524e4d09ba07b79634e1033f" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 45109 } } peers { permanent_uuid: "0def6aa011f14329ad489b2311ac46a1" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 39991 } }
I20260928 06:37:22.448904 4739 raft_consensus.cc:348] T 5b9518f1d0f144af8bead119dcf7770a P 0def6aa011f14329ad489b2311ac46a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e96ff6bffdb4b0983159e3b967752f5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 44713 } } peers { permanent_uuid: "c8627e98524e4d09ba07b79634e1033f" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 45109 } } peers { permanent_uuid: "0def6aa011f14329ad489b2311ac46a1" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 39991 } }
I20260928 06:37:22.449100 4739 raft_consensus.cc:374] T 5b9518f1d0f144af8bead119dcf7770a P 0def6aa011f14329ad489b2311ac46a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:37:22.449144 4592 heartbeater.cc:503] Master 127.16.157.126:35183 was elected leader, sending a full tablet report...
I20260928 06:37:22.449152 4739 raft_consensus.cc:729] T 5b9518f1d0f144af8bead119dcf7770a P 0def6aa011f14329ad489b2311ac46a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0def6aa011f14329ad489b2311ac46a1, State: Initialized, Role: FOLLOWER
I20260928 06:37:22.449268 4739 consensus_queue.cc:260] T 5b9518f1d0f144af8bead119dcf7770a P 0def6aa011f14329ad489b2311ac46a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e96ff6bffdb4b0983159e3b967752f5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 44713 } } peers { permanent_uuid: "c8627e98524e4d09ba07b79634e1033f" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 45109 } } peers { permanent_uuid: "0def6aa011f14329ad489b2311ac46a1" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 39991 } }
I20260928 06:37:22.449442 4737 ts_tablet_manager.cc:1428] T 5b9518f1d0f144af8bead119dcf7770a P c8627e98524e4d09ba07b79634e1033f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:37:22.449553 4739 ts_tablet_manager.cc:1428] T 5b9518f1d0f144af8bead119dcf7770a P 0def6aa011f14329ad489b2311ac46a1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:37:22.449718 4461 heartbeater.cc:503] Master 127.16.157.126:35183 was elected leader, sending a full tablet report...
I20260928 06:37:22.449955 4738 tablet_bootstrap.cc:492] T 5b9518f1d0f144af8bead119dcf7770a P 8e96ff6bffdb4b0983159e3b967752f5: No bootstrap required, opened a new log
I20260928 06:37:22.450021 4738 ts_tablet_manager.cc:1397] T 5b9518f1d0f144af8bead119dcf7770a P 8e96ff6bffdb4b0983159e3b967752f5: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260928 06:37:22.452258 4738 raft_consensus.cc:348] T 5b9518f1d0f144af8bead119dcf7770a P 8e96ff6bffdb4b0983159e3b967752f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e96ff6bffdb4b0983159e3b967752f5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 44713 } } peers { permanent_uuid: "c8627e98524e4d09ba07b79634e1033f" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 45109 } } peers { permanent_uuid: "0def6aa011f14329ad489b2311ac46a1" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 39991 } }
I20260928 06:37:22.452407 4738 raft_consensus.cc:374] T 5b9518f1d0f144af8bead119dcf7770a P 8e96ff6bffdb4b0983159e3b967752f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:37:22.452451 4738 raft_consensus.cc:729] T 5b9518f1d0f144af8bead119dcf7770a P 8e96ff6bffdb4b0983159e3b967752f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e96ff6bffdb4b0983159e3b967752f5, State: Initialized, Role: FOLLOWER
I20260928 06:37:22.452567 4738 consensus_queue.cc:260] T 5b9518f1d0f144af8bead119dcf7770a P 8e96ff6bffdb4b0983159e3b967752f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e96ff6bffdb4b0983159e3b967752f5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 44713 } } peers { permanent_uuid: "c8627e98524e4d09ba07b79634e1033f" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 45109 } } peers { permanent_uuid: "0def6aa011f14329ad489b2311ac46a1" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 39991 } }
I20260928 06:37:22.452822 4738 ts_tablet_manager.cc:1428] T 5b9518f1d0f144af8bead119dcf7770a P 8e96ff6bffdb4b0983159e3b967752f5: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 06:37:22.452917 4723 heartbeater.cc:503] Master 127.16.157.126:35183 was elected leader, sending a full tablet report...
W20260928 06:37:22.533965 4593 tablet.cc:2369] T 5b9518f1d0f144af8bead119dcf7770a P c8627e98524e4d09ba07b79634e1033f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:37:22.582613 4743 raft_consensus.cc:482] T 5b9518f1d0f144af8bead119dcf7770a P c8627e98524e4d09ba07b79634e1033f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:37:22.582738 4743 raft_consensus.cc:504] T 5b9518f1d0f144af8bead119dcf7770a P c8627e98524e4d09ba07b79634e1033f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e96ff6bffdb4b0983159e3b967752f5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 44713 } } peers { permanent_uuid: "c8627e98524e4d09ba07b79634e1033f" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 45109 } } peers { permanent_uuid: "0def6aa011f14329ad489b2311ac46a1" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 39991 } }
I20260928 06:37:22.583068 4743 leader_election.cc:288] T 5b9518f1d0f144af8bead119dcf7770a P c8627e98524e4d09ba07b79634e1033f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e96ff6bffdb4b0983159e3b967752f5 (127.16.157.67:44713), 0def6aa011f14329ad489b2311ac46a1 (127.16.157.65:39991)
I20260928 06:37:22.586798 4594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:22.583251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.67:44713 (local address 127.16.157.66:53737)
0928 06:37:22.583314 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:22.583327 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:22.583397 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:22.583560 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:22.583561 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:22.583563 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:37:22.583644 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:22.583646 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:22.584586 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:22.584587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:22.585680 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:22.585683 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:22.586497 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:22.586499 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:22.586585 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:22.586590 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:37:22.586651 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 06:37:22.586702 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":63}
I20260928 06:37:22.586798 4733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:22.583363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.66:53737 (local address 127.16.157.67:44713)
0928 06:37:22.583428 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:22.583430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:22.583442 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:37:22.583476 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:22.583476 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:22.583485 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:22.583549 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:37:22.583718 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:22.584497 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:22.585782 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:22.586366 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:22.586447 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:22.586458 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:22.586664 (+ 206us) server_negotiation.cc:299] Negotiation successful
0928 06:37:22.586719 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 06:37:22.587236 4736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:22.583446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.157.66:36935 (local address 127.16.157.65:39991)
0928 06:37:22.583490 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:22.583493 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:22.583618 (+ 125us) server_negotiation.cc:407] Connection header received
0928 06:37:22.583656 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:22.583657 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:22.583668 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:22.583750 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:37:22.583888 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:22.584562 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:22.585959 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:22.586547 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:22.586637 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:22.586646 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:22.587127 (+ 481us) server_negotiation.cc:299] Negotiation successful
0928 06:37:22.587176 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:37:22.587237 4746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:22.583361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.65:39991 (local address 127.16.157.66:36935)
0928 06:37:22.583534 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:22.583544 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:22.583602 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:22.583769 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:22.583770 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:22.583771 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:37:22.583819 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:22.583819 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:22.584766 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:22.584767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:22.585846 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:22.585850 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:22.586948 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:22.586949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:22.587038 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:22.587042 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:37:22.587109 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 06:37:22.587150 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":1,"thread_start_us":111,"threads_started":1}
I20260928 06:37:22.587227 4677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b9518f1d0f144af8bead119dcf7770a" candidate_uuid: "c8627e98524e4d09ba07b79634e1033f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e96ff6bffdb4b0983159e3b967752f5" is_pre_election: true
I20260928 06:37:22.587383 4677 raft_consensus.cc:2454] T 5b9518f1d0f144af8bead119dcf7770a P 8e96ff6bffdb4b0983159e3b967752f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8627e98524e4d09ba07b79634e1033f in term 0.
I20260928 06:37:22.587704 4481 leader_election.cc:302] T 5b9518f1d0f144af8bead119dcf7770a P c8627e98524e4d09ba07b79634e1033f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e96ff6bffdb4b0983159e3b967752f5, c8627e98524e4d09ba07b79634e1033f; no voters:
I20260928 06:37:22.587865 4743 raft_consensus.cc:2781] T 5b9518f1d0f144af8bead119dcf7770a P c8627e98524e4d09ba07b79634e1033f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:37:22.587823 4415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b9518f1d0f144af8bead119dcf7770a" candidate_uuid: "c8627e98524e4d09ba07b79634e1033f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0def6aa011f14329ad489b2311ac46a1" is_pre_election: true
I20260928 06:37:22.587922 4743 raft_consensus.cc:482] T 5b9518f1d0f144af8bead119dcf7770a P c8627e98524e4d09ba07b79634e1033f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:37:22.587950 4743 raft_consensus.cc:3037] T 5b9518f1d0f144af8bead119dcf7770a P c8627e98524e4d09ba07b79634e1033f [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:37:22.587996 4415 raft_consensus.cc:2454] T 5b9518f1d0f144af8bead119dcf7770a P 0def6aa011f14329ad489b2311ac46a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8627e98524e4d09ba07b79634e1033f in term 0.
I20260928 06:37:22.588948 4743 raft_consensus.cc:504] T 5b9518f1d0f144af8bead119dcf7770a P c8627e98524e4d09ba07b79634e1033f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e96ff6bffdb4b0983159e3b967752f5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 44713 } } peers { permanent_uuid: "c8627e98524e4d09ba07b79634e1033f" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 45109 } } peers { permanent_uuid: "0def6aa011f14329ad489b2311ac46a1" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 39991 } }
I20260928 06:37:22.589097 4743 leader_election.cc:288] T 5b9518f1d0f144af8bead119dcf7770a P c8627e98524e4d09ba07b79634e1033f [CANDIDATE]: Term 1 election: Requested vote from peers 8e96ff6bffdb4b0983159e3b967752f5 (127.16.157.67:44713), 0def6aa011f14329ad489b2311ac46a1 (127.16.157.65:39991)
I20260928 06:37:22.589278 4415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b9518f1d0f144af8bead119dcf7770a" candidate_uuid: "c8627e98524e4d09ba07b79634e1033f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0def6aa011f14329ad489b2311ac46a1"
I20260928 06:37:22.589280 4677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b9518f1d0f144af8bead119dcf7770a" candidate_uuid: "c8627e98524e4d09ba07b79634e1033f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e96ff6bffdb4b0983159e3b967752f5"
I20260928 06:37:22.589389 4677 raft_consensus.cc:3037] T 5b9518f1d0f144af8bead119dcf7770a P 8e96ff6bffdb4b0983159e3b967752f5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:37:22.589386 4415 raft_consensus.cc:3037] T 5b9518f1d0f144af8bead119dcf7770a P 0def6aa011f14329ad489b2311ac46a1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:37:22.590207 4677 raft_consensus.cc:2454] T 5b9518f1d0f144af8bead119dcf7770a P 8e96ff6bffdb4b0983159e3b967752f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8627e98524e4d09ba07b79634e1033f in term 1.
I20260928 06:37:22.590232 4415 raft_consensus.cc:2454] T 5b9518f1d0f144af8bead119dcf7770a P 0def6aa011f14329ad489b2311ac46a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8627e98524e4d09ba07b79634e1033f in term 1.
I20260928 06:37:22.590389 4481 leader_election.cc:302] T 5b9518f1d0f144af8bead119dcf7770a P c8627e98524e4d09ba07b79634e1033f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0def6aa011f14329ad489b2311ac46a1, c8627e98524e4d09ba07b79634e1033f; no voters:
I20260928 06:37:22.590508 4743 raft_consensus.cc:2781] T 5b9518f1d0f144af8bead119dcf7770a P c8627e98524e4d09ba07b79634e1033f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:37:22.590646 4743 raft_consensus.cc:686] T 5b9518f1d0f144af8bead119dcf7770a P c8627e98524e4d09ba07b79634e1033f [term 1 LEADER]: Becoming Leader. State: Replica: c8627e98524e4d09ba07b79634e1033f, State: Running, Role: LEADER
I20260928 06:37:22.590880 4743 consensus_queue.cc:237] T 5b9518f1d0f144af8bead119dcf7770a P c8627e98524e4d09ba07b79634e1033f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e96ff6bffdb4b0983159e3b967752f5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 44713 } } peers { permanent_uuid: "c8627e98524e4d09ba07b79634e1033f" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 45109 } } peers { permanent_uuid: "0def6aa011f14329ad489b2311ac46a1" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 39991 } }
I20260928 06:37:22.591724 4273 catalog_manager.cc:5495] T 5b9518f1d0f144af8bead119dcf7770a P c8627e98524e4d09ba07b79634e1033f reported cstate change: term changed from 0 to 1, leader changed from <none> to c8627e98524e4d09ba07b79634e1033f (127.16.157.66). New cstate: current_term: 1 leader_uuid: "c8627e98524e4d09ba07b79634e1033f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e96ff6bffdb4b0983159e3b967752f5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 44713 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8627e98524e4d09ba07b79634e1033f" member_type: VOTER last_known_addr { host: "127.16.157.66" port: 45109 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0def6aa011f14329ad489b2311ac46a1" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 39991 } health_report { overall_health: UNKNOWN } } }
I20260928 06:37:22.625195 4734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:22.622743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35506 (local address 127.16.157.66:45109)
0928 06:37:22.622809 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:22.622811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:22.622820 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:37:22.622887 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:22.622887 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:22.622912 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:22.622986 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:37:22.623113 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:22.623611 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:22.624348 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:22.624604 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:22.624679 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:22.624831 (+ 152us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:37:22.624950 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:37:22.625035 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:22.625074 (+ 39us) server_negotiation.cc:299] Negotiation successful
0928 06:37:22.625093 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 06:37:22.627075 4415 raft_consensus.cc:1261] T 5b9518f1d0f144af8bead119dcf7770a P 0def6aa011f14329ad489b2311ac46a1 [term 1 FOLLOWER]: Refusing update from remote peer c8627e98524e4d09ba07b79634e1033f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:37:22.627120 4677 raft_consensus.cc:1261] T 5b9518f1d0f144af8bead119dcf7770a P 8e96ff6bffdb4b0983159e3b967752f5 [term 1 FOLLOWER]: Refusing update from remote peer c8627e98524e4d09ba07b79634e1033f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:37:22.627436 4743 consensus_queue.cc:1035] T 5b9518f1d0f144af8bead119dcf7770a P c8627e98524e4d09ba07b79634e1033f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0def6aa011f14329ad489b2311ac46a1" member_type: VOTER last_known_addr { host: "127.16.157.65" port: 39991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:37:22.627583 4747 consensus_queue.cc:1035] T 5b9518f1d0f144af8bead119dcf7770a P c8627e98524e4d09ba07b79634e1033f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e96ff6bffdb4b0983159e3b967752f5" member_type: VOTER last_known_addr { host: "127.16.157.67" port: 44713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:37:22.632130 4273 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56526:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:37:22.632229 4273 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56526:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:37:22.632973 4273 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6843ec1355dd404787658d74609e90df: Sending DeleteTablet for 3 replicas of tablet 5b9518f1d0f144af8bead119dcf7770a
I20260928 06:37:22.633307 4526 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b9518f1d0f144af8bead119dcf7770a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:37:22 UTC) from {username='slave'} at 127.0.0.1:35494
I20260928 06:37:22.633333 4395 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b9518f1d0f144af8bead119dcf7770a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:37:22 UTC) from {username='slave'} at 127.0.0.1:46362
I20260928 06:37:22.633611 4657 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b9518f1d0f144af8bead119dcf7770a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:37:22 UTC) from {username='slave'} at 127.0.0.1:38752
I20260928 06:37:22.633841 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 4333
I20260928 06:37:22.633857 4762 tablet_replica.cc:330] T 5b9518f1d0f144af8bead119dcf7770a P 8e96ff6bffdb4b0983159e3b967752f5: stopping tablet replica
I20260928 06:37:22.633950 4762 raft_consensus.cc:2229] T 5b9518f1d0f144af8bead119dcf7770a P 8e96ff6bffdb4b0983159e3b967752f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:37:22.634038 4762 raft_consensus.cc:2258] T 5b9518f1d0f144af8bead119dcf7770a P 8e96ff6bffdb4b0983159e3b967752f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:37:22.635154 4760 tablet_replica.cc:330] T 5b9518f1d0f144af8bead119dcf7770a P c8627e98524e4d09ba07b79634e1033f: stopping tablet replica
I20260928 06:37:22.635246 4760 raft_consensus.cc:2229] T 5b9518f1d0f144af8bead119dcf7770a P c8627e98524e4d09ba07b79634e1033f [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:37:22.635331 4760 raft_consensus.cc:2258] T 5b9518f1d0f144af8bead119dcf7770a P c8627e98524e4d09ba07b79634e1033f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:37:22.635717 4760 ts_tablet_manager.cc:1905] T 5b9518f1d0f144af8bead119dcf7770a P c8627e98524e4d09ba07b79634e1033f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:37:22.637816 4760 ts_tablet_manager.cc:1918] T 5b9518f1d0f144af8bead119dcf7770a P c8627e98524e4d09ba07b79634e1033f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:37:22.637894 4760 log.cc:1196] T 5b9518f1d0f144af8bead119dcf7770a P c8627e98524e4d09ba07b79634e1033f: Deleting WAL directory at /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577196946045-17013-0/minicluster-data/ts-1/wal/wals/5b9518f1d0f144af8bead119dcf7770a
I20260928 06:37:22.638214 4760 ts_tablet_manager.cc:1939] T 5b9518f1d0f144af8bead119dcf7770a P c8627e98524e4d09ba07b79634e1033f: Deleting consensus metadata
I20260928 06:37:22.638381 4762 ts_tablet_manager.cc:1905] T 5b9518f1d0f144af8bead119dcf7770a P 8e96ff6bffdb4b0983159e3b967752f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:37:22.638607 4260 catalog_manager.cc:4841] TS c8627e98524e4d09ba07b79634e1033f (127.16.157.66:45109): tablet 5b9518f1d0f144af8bead119dcf7770a (table test-table [id=9444b4d687a1446a9a42c5e6bbc24578]) successfully deleted
I20260928 06:37:22.639765 4762 ts_tablet_manager.cc:1918] T 5b9518f1d0f144af8bead119dcf7770a P 8e96ff6bffdb4b0983159e3b967752f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:37:22.639848 4762 log.cc:1196] T 5b9518f1d0f144af8bead119dcf7770a P 8e96ff6bffdb4b0983159e3b967752f5: Deleting WAL directory at /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577196946045-17013-0/minicluster-data/ts-2/wal/wals/5b9518f1d0f144af8bead119dcf7770a
I20260928 06:37:22.640112 4762 ts_tablet_manager.cc:1939] T 5b9518f1d0f144af8bead119dcf7770a P 8e96ff6bffdb4b0983159e3b967752f5: Deleting consensus metadata
I20260928 06:37:22.640534 4259 catalog_manager.cc:4841] TS 8e96ff6bffdb4b0983159e3b967752f5 (127.16.157.67:44713): tablet 5b9518f1d0f144af8bead119dcf7770a (table test-table [id=9444b4d687a1446a9a42c5e6bbc24578]) successfully deleted
W20260928 06:37:22.642560 4259 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.157.65:39991 (error 108)
I20260928 06:37:22.642812 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 4464
I20260928 06:37:22.642983 4731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:22.642876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.65:39991 (local address 127.0.0.1:46366)
0928 06:37:22.642925 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:22.642956 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.65:39991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 06:37:22.644469 4259 catalog_manager.cc:4584] TS 0def6aa011f14329ad489b2311ac46a1 (127.16.157.65:39991): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5b9518f1d0f144af8bead119dcf7770a: Network error: Client connection negotiation failed: client connection to 127.16.157.65:39991: connect: Connection refused (error 111)
I20260928 06:37:22.648185 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 4595
I20260928 06:37:22.652753 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 4242
2026-09-28T06:37:22Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (822 ms)
[----------] 6 tests from EncryptionPolicyTest (8529 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577196946045-17013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:37:22 dist-test-slave-rrj5 krb5kdc[4766](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:37:22 dist-test-slave-rrj5 krb5kdc[4766](info): set up 2 sockets
Sep 28 06:37:22 dist-test-slave-rrj5 krb5kdc[4766](info): commencing operation
krb5kdc: starting...
W20260928 06:37:23.670454 17013 mini_kdc.cc:121] Time spent starting KDC: real 0.998s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Sep 28 06:37:23 dist-test-slave-rrj5 krb5kdc[4766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:37:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:37:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577196946045-17013-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577196946045-17013-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:37:23.785153 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577196946045-17013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577196946045-17013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577196946045-17013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34323
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577196946045-17013-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34323
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:37:23.867084 4779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:23.867341 4779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:23.867383 4779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:23.868654 4779 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:37:23.868734 4779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:23.868762 4779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:23.868793 4779 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:37:23.868826 4779 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:37:23.870322 4779 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577196946045-17013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577196946045-17013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34323
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577196946045-17013-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577196946045-17013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:37:23.870594 4779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:23.870807 4779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:23.875669 4787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:23.875674 4785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:23.875717 4784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:23.875999 4779 server_base.cc:1034] running on GCE node
I20260928 06:37:23.876236 4779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:23.876520 4779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:23.877652 4779 hybrid_clock.cc:648] HybridClock initialized: now 1790577443877633 us; error 39 us; skew 500 ppm
Sep 28 06:37:23 dist-test-slave-rrj5 krb5kdc[4766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:23.880012 4779 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:37:23.880420 4779 webserver.cc:460] Webserver started at http://127.0.0.1:35797/ using document root <none> and password file <none>
I20260928 06:37:23.880661 4779 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:23.880729 4779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:23.880892 4779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:37:23.881973 4779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577196946045-17013-0/minicluster-data/master-0/data/instance:
uuid: "22c62630a1a14c38a1cd6412fd331703"
format_stamp: "Formatted at 2026-09-28 06:37:23 on dist-test-slave-rrj5"
server_key: "c466f0117d9c8895beff3339e059a169"
server_key_iv: "9f7358bd47391d59afb688bb60ce997f"
server_key_version: "encryptionkey@0"
I20260928 06:37:23.882321 4779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577196946045-17013-0/minicluster-data/master-0/wal/instance:
uuid: "22c62630a1a14c38a1cd6412fd331703"
format_stamp: "Formatted at 2026-09-28 06:37:23 on dist-test-slave-rrj5"
server_key: "c466f0117d9c8895beff3339e059a169"
server_key_iv: "9f7358bd47391d59afb688bb60ce997f"
server_key_version: "encryptionkey@0"
I20260928 06:37:23.883677 4779 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:37:23.884469 4794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:23.884683 4779 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:37:23.884774 4779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577196946045-17013-0/minicluster-data/master-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577196946045-17013-0/minicluster-data/master-0/wal
uuid: "22c62630a1a14c38a1cd6412fd331703"
format_stamp: "Formatted at 2026-09-28 06:37:23 on dist-test-slave-rrj5"
server_key: "c466f0117d9c8895beff3339e059a169"
server_key_iv: "9f7358bd47391d59afb688bb60ce997f"
server_key_version: "encryptionkey@0"
I20260928 06:37:23.884855 4779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577196946045-17013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577196946045-17013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577196946045-17013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:23.921828 4779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:23.923951 4779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:23.924121 4779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:23.928433 4779 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34323
I20260928 06:37:23.928471 4846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34323 every 8 connection(s)
I20260928 06:37:23.928855 4779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
I20260928 06:37:23.929874 4847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:37:23.929936 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 4779
I20260928 06:37:23.930042 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577196946045-17013-0/minicluster-data/master-0/wal/instance
I20260928 06:37:23.930277 17013 external_mini_cluster.cc:1428] Setting key ee4cda3b57b6a2bf94d51913ca738b43
I20260928 06:37:23.933251 4847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22c62630a1a14c38a1cd6412fd331703: Bootstrap starting.
I20260928 06:37:23.934000 4847 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22c62630a1a14c38a1cd6412fd331703: Neither blocks nor log segments found. Creating new log.
I20260928 06:37:23.934304 4847 log.cc:824] T 00000000000000000000000000000000 P 22c62630a1a14c38a1cd6412fd331703: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:23.935038 4847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22c62630a1a14c38a1cd6412fd331703: No bootstrap required, opened a new log
Sep 28 06:37:23 dist-test-slave-rrj5 krb5kdc[4766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 06:37:23.937317 4847 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22c62630a1a14c38a1cd6412fd331703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22c62630a1a14c38a1cd6412fd331703" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34323 } }
I20260928 06:37:23.937495 4847 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22c62630a1a14c38a1cd6412fd331703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:37:23.937548 4847 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22c62630a1a14c38a1cd6412fd331703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22c62630a1a14c38a1cd6412fd331703, State: Initialized, Role: FOLLOWER
I20260928 06:37:23.937722 4847 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22c62630a1a14c38a1cd6412fd331703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22c62630a1a14c38a1cd6412fd331703" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34323 } }
I20260928 06:37:23.937813 4847 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22c62630a1a14c38a1cd6412fd331703 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:37:23.937852 4847 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22c62630a1a14c38a1cd6412fd331703 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:37:23.937909 4847 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22c62630a1a14c38a1cd6412fd331703 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:37:23.938628 4847 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22c62630a1a14c38a1cd6412fd331703 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22c62630a1a14c38a1cd6412fd331703" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34323 } }
I20260928 06:37:23.938759 4847 leader_election.cc:302] T 00000000000000000000000000000000 P 22c62630a1a14c38a1cd6412fd331703 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22c62630a1a14c38a1cd6412fd331703; no voters:
I20260928 06:37:23.938995 4847 leader_election.cc:288] T 00000000000000000000000000000000 P 22c62630a1a14c38a1cd6412fd331703 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:37:23.939041 4852 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22c62630a1a14c38a1cd6412fd331703 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:37:23.939179 4852 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22c62630a1a14c38a1cd6412fd331703 [term 1 LEADER]: Becoming Leader. State: Replica: 22c62630a1a14c38a1cd6412fd331703, State: Running, Role: LEADER
I20260928 06:37:23.939311 4847 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22c62630a1a14c38a1cd6412fd331703 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:37:23.939360 4852 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22c62630a1a14c38a1cd6412fd331703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22c62630a1a14c38a1cd6412fd331703" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34323 } }
I20260928 06:37:23.939906 4854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22c62630a1a14c38a1cd6412fd331703 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22c62630a1a14c38a1cd6412fd331703. Latest consensus state: current_term: 1 leader_uuid: "22c62630a1a14c38a1cd6412fd331703" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22c62630a1a14c38a1cd6412fd331703" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34323 } } }
I20260928 06:37:23.940016 4854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22c62630a1a14c38a1cd6412fd331703 [sys.catalog]: This master's current role is: LEADER
I20260928 06:37:23.940696 4859 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:37:23.941214 4859 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:37:23.941848 4850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:23.930827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57313 (local address 127.0.0.1:34323)
0928 06:37:23.931004 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:23.931011 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:23.931037 (+ 26us) server_negotiation.cc:407] Connection header received
0928 06:37:23.931511 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:23.931518 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:23.931563 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:23.931681 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:23.932489 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:23.933419 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:23.934568 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:23.935058 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:23.935189 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:23.937955 (+ 2766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:23.937959 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:23.937959 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:23.937972 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:23.940046 (+ 2074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:23.940518 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:23.940521 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:23.940521 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:23.940625 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:23.940886 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:23.940887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:23.940887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:23.941184 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:23.941348 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:23.941477 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 06:37:23.941605 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":13,"server-negotiator.queue_time_us":124,"thread_start_us":88,"threads_started":1}
I20260928 06:37:23.942396 4853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22c62630a1a14c38a1cd6412fd331703 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22c62630a1a14c38a1cd6412fd331703" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22c62630a1a14c38a1cd6412fd331703" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34323 } } }
I20260928 06:37:23.942462 4853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22c62630a1a14c38a1cd6412fd331703 [sys.catalog]: This master's current role is: LEADER
I20260928 06:37:23.942940 17013 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:37:23.943113 4859 catalog_manager.cc:1347] Generated new cluster ID: dd43d6bfc83143039dacadcb8593fc38
I20260928 06:37:23.943173 4859 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:37:23.949548 4859 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:37:23.950313 4859 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:37:23.950641 4850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:23.944272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35549 (local address 127.0.0.1:34323)
0928 06:37:23.944310 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:23.944311 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:23.944426 (+ 115us) server_negotiation.cc:407] Connection header received
0928 06:37:23.944486 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:23.944486 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:23.944496 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:23.944539 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:23.945048 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:23.945544 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:23.946306 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:23.946556 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:23.946650 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:23.947761 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:23.947762 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:23.947762 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:23.947769 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:23.949310 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:23.949751 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:23.949752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:23.949752 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:23.949813 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:23.950040 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:23.950041 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:23.950042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:23.950216 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:23.950353 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:23.950475 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 06:37:23.950536 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":18}
I20260928 06:37:23.955580 4859 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22c62630a1a14c38a1cd6412fd331703: Generated new TSK 0
I20260928 06:37:23.955768 4859 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:37:23.960788 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 4779
W20260928 06:37:23.965271 4764 connection.cc:535] client connection to 127.0.0.1:34323 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34323): Connection reset by peer (error 104)
2026-09-28T06:37:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1306 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577196946045-17013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:37:24 dist-test-slave-rrj5 krb5kdc[4878](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:37:24 dist-test-slave-rrj5 krb5kdc[4878](info): set up 2 sockets
Sep 28 06:37:24 dist-test-slave-rrj5 krb5kdc[4878](info): commencing operation
krb5kdc: starting...
W20260928 06:37:24.982534 17013 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Sep 28 06:37:25 dist-test-slave-rrj5 krb5kdc[4878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:37:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:37:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577196946045-17013-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577196946045-17013-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:37:25.101209 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577196946045-17013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577196946045-17013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577196946045-17013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41929
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577196946045-17013-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41929
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:37:25.184087 4891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:25.184345 4891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:25.184386 4891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:25.185667 4891 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:37:25.185734 4891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:25.185757 4891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:25.185779 4891 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:37:25.185806 4891 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:37:25.187368 4891 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577196946045-17013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577196946045-17013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41929
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577196946045-17013-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577196946045-17013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:37:25.187664 4891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:25.187913 4891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:25.192916 4897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:25.192924 4899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:25.192960 4896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:25.193130 4891 server_base.cc:1034] running on GCE node
I20260928 06:37:25.193460 4891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:25.193897 4891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:25.195079 4891 hybrid_clock.cc:648] HybridClock initialized: now 1790577445195051 us; error 45 us; skew 500 ppm
Sep 28 06:37:25 dist-test-slave-rrj5 krb5kdc[4878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577445, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:25.201104 4891 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:37:25.201531 4891 webserver.cc:460] Webserver started at http://127.0.0.1:40799/ using document root <none> and password file <none>
I20260928 06:37:25.201824 4891 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:25.201874 4891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:25.201964 4891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:37:25.203002 4891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577196946045-17013-0/minicluster-data/master-0/data/instance:
uuid: "2aaed62e2ae445f48ca76a52c104c876"
format_stamp: "Formatted at 2026-09-28 06:37:25 on dist-test-slave-rrj5"
server_key: "5f6e138db721399fad019d71ba4102fb"
server_key_iv: "7968def3ce09327b6b2873f4436bcb8f"
server_key_version: "encryptionkey@0"
I20260928 06:37:25.203299 4891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577196946045-17013-0/minicluster-data/master-0/wal/instance:
uuid: "2aaed62e2ae445f48ca76a52c104c876"
format_stamp: "Formatted at 2026-09-28 06:37:25 on dist-test-slave-rrj5"
server_key: "5f6e138db721399fad019d71ba4102fb"
server_key_iv: "7968def3ce09327b6b2873f4436bcb8f"
server_key_version: "encryptionkey@0"
I20260928 06:37:25.204581 4891 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:37:25.205293 4906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:25.205474 4891 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:37:25.205617 4891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577196946045-17013-0/minicluster-data/master-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577196946045-17013-0/minicluster-data/master-0/wal
uuid: "2aaed62e2ae445f48ca76a52c104c876"
format_stamp: "Formatted at 2026-09-28 06:37:25 on dist-test-slave-rrj5"
server_key: "5f6e138db721399fad019d71ba4102fb"
server_key_iv: "7968def3ce09327b6b2873f4436bcb8f"
server_key_version: "encryptionkey@0"
I20260928 06:37:25.205713 4891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577196946045-17013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577196946045-17013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577196946045-17013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:25.234764 4891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:25.237190 4891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:25.237377 4891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:25.241641 4891 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41929
I20260928 06:37:25.241693 4958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41929 every 8 connection(s)
I20260928 06:37:25.242084 4891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
I20260928 06:37:25.243018 4959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:37:25.245940 4959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2aaed62e2ae445f48ca76a52c104c876: Bootstrap starting.
I20260928 06:37:25.246402 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 4891
I20260928 06:37:25.246548 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577196946045-17013-0/minicluster-data/master-0/wal/instance
I20260928 06:37:25.246668 4959 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2aaed62e2ae445f48ca76a52c104c876: Neither blocks nor log segments found. Creating new log.
I20260928 06:37:25.246794 17013 external_mini_cluster.cc:1428] Setting key 754439a79d0b13b5872bb75b906b28d1
I20260928 06:37:25.247004 4959 log.cc:824] T 00000000000000000000000000000000 P 2aaed62e2ae445f48ca76a52c104c876: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:25.247809 4959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2aaed62e2ae445f48ca76a52c104c876: No bootstrap required, opened a new log
I20260928 06:37:25.250125 4959 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2aaed62e2ae445f48ca76a52c104c876 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aaed62e2ae445f48ca76a52c104c876" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41929 } }
I20260928 06:37:25.250319 4959 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2aaed62e2ae445f48ca76a52c104c876 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:37:25.250353 4959 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2aaed62e2ae445f48ca76a52c104c876 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aaed62e2ae445f48ca76a52c104c876, State: Initialized, Role: FOLLOWER
I20260928 06:37:25.250521 4959 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2aaed62e2ae445f48ca76a52c104c876 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aaed62e2ae445f48ca76a52c104c876" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41929 } }
I20260928 06:37:25.250628 4959 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2aaed62e2ae445f48ca76a52c104c876 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:37:25.250681 4959 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2aaed62e2ae445f48ca76a52c104c876 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:37:25.250738 4959 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2aaed62e2ae445f48ca76a52c104c876 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:37:25.251422 4959 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2aaed62e2ae445f48ca76a52c104c876 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aaed62e2ae445f48ca76a52c104c876" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41929 } }
I20260928 06:37:25.251545 4959 leader_election.cc:302] T 00000000000000000000000000000000 P 2aaed62e2ae445f48ca76a52c104c876 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2aaed62e2ae445f48ca76a52c104c876; no voters:
I20260928 06:37:25.251713 4959 leader_election.cc:288] T 00000000000000000000000000000000 P 2aaed62e2ae445f48ca76a52c104c876 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:37:25.251788 4964 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2aaed62e2ae445f48ca76a52c104c876 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:37:25.251943 4964 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2aaed62e2ae445f48ca76a52c104c876 [term 1 LEADER]: Becoming Leader. State: Replica: 2aaed62e2ae445f48ca76a52c104c876, State: Running, Role: LEADER
I20260928 06:37:25.252035 4964 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2aaed62e2ae445f48ca76a52c104c876 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aaed62e2ae445f48ca76a52c104c876" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41929 } }
I20260928 06:37:25.252058 4959 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2aaed62e2ae445f48ca76a52c104c876 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:37:25.252430 4965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2aaed62e2ae445f48ca76a52c104c876 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2aaed62e2ae445f48ca76a52c104c876. Latest consensus state: current_term: 1 leader_uuid: "2aaed62e2ae445f48ca76a52c104c876" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aaed62e2ae445f48ca76a52c104c876" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41929 } } }
I20260928 06:37:25.253237 4965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2aaed62e2ae445f48ca76a52c104c876 [sys.catalog]: This master's current role is: LEADER
I20260928 06:37:25.252432 4966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2aaed62e2ae445f48ca76a52c104c876 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2aaed62e2ae445f48ca76a52c104c876" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aaed62e2ae445f48ca76a52c104c876" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41929 } } }
I20260928 06:37:25.253610 4966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2aaed62e2ae445f48ca76a52c104c876 [sys.catalog]: This master's current role is: LEADER
I20260928 06:37:25.253921 4963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:25.247343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43231 (local address 127.0.0.1:41929)
0928 06:37:25.247541 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:25.247551 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:25.247588 (+ 37us) server_negotiation.cc:407] Connection header received
0928 06:37:25.248081 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:25.248089 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:25.248146 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:25.248262 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:25.248520 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:25.249377 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:25.250570 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:25.251108 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:25.251190 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:25.252814 (+ 1624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:25.252819 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:25.252819 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:37:25.253049 (+ 230us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:25.253093 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:37:25.253111 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:25.253204 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:25.253491 (+ 287us) server_negotiation.cc:299] Negotiation successful
0928 06:37:25.253528 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":112,"threads_started":1}
I20260928 06:37:25.253556 4976 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:37:25.254455 4976 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:37:25.254969 17013 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:37:25.256105 4976 catalog_manager.cc:1347] Generated new cluster ID: 315e7aac5d2940ebbbaf9c189b5948af
I20260928 06:37:25.256162 4976 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:37:25.260751 4963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:25.256461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47859 (local address 127.0.0.1:41929)
0928 06:37:25.256530 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:25.256532 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:25.256557 (+ 25us) server_negotiation.cc:407] Connection header received
0928 06:37:25.256635 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:25.256635 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:25.256644 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:25.256730 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:25.256904 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:25.257836 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:25.259045 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:25.259460 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:25.259525 (+ 65us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:25.260393 (+ 868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:25.260395 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:25.260395 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:37:25.260400 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:25.260414 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:37:25.260418 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:25.260506 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:25.260591 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 06:37:25.260601 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:37:25.264040 4976 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:37:25.264634 4976 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:37:25.270486 4976 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2aaed62e2ae445f48ca76a52c104c876: Generated new TSK 0
I20260928 06:37:25.270706 4976 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:37:25.280236 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 4891
W20260928 06:37:25.284906 4876 connection.cc:535] client connection to 127.0.0.1:41929 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41929): Connection reset by peer (error 104)
2026-09-28T06:37:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1319 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577196946045-17013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:37:25 dist-test-slave-rrj5 krb5kdc[4990](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:37:25 dist-test-slave-rrj5 krb5kdc[4990](info): set up 2 sockets
Sep 28 06:37:25 dist-test-slave-rrj5 krb5kdc[4990](info): commencing operation
krb5kdc: starting...
W20260928 06:37:26.318490 17013 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Sep 28 06:37:26 dist-test-slave-rrj5 krb5kdc[4990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:37:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:37:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577196946045-17013-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577196946045-17013-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:37:26.436913 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577196946045-17013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577196946045-17013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577196946045-17013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35551
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577196946045-17013-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35551
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:37:26.515554 5003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:26.515793 5003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:26.515831 5003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:26.516949 5003 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:37:26.517020 5003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:26.517045 5003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:26.517067 5003 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:37:26.517097 5003 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:37:26.518540 5003 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577196946045-17013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577196946045-17013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35551
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577196946045-17013-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577196946045-17013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:37:26.518816 5003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:26.519034 5003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:26.523773 5011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:26.523765 5009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:26.523764 5008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:26.524000 5003 server_base.cc:1034] running on GCE node
I20260928 06:37:26.524215 5003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:26.524441 5003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:26.525632 5003 hybrid_clock.cc:648] HybridClock initialized: now 1790577446525606 us; error 40 us; skew 500 ppm
Sep 28 06:37:26 dist-test-slave-rrj5 krb5kdc[4990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577446, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:26.527922 5003 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:37:26.528288 5003 webserver.cc:460] Webserver started at http://127.0.0.1:41503/ using document root <none> and password file <none>
I20260928 06:37:26.528481 5003 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:26.528532 5003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:26.528613 5003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:37:26.529505 5003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577196946045-17013-0/minicluster-data/master-0/data/instance:
uuid: "198ef733e82e4f97be5b8a09af844e3e"
format_stamp: "Formatted at 2026-09-28 06:37:26 on dist-test-slave-rrj5"
server_key: "d5d8b85b54391568e48c2d22346fe965"
server_key_iv: "115c857759d82c153bd947a81ab35a88"
server_key_version: "encryptionkey@0"
I20260928 06:37:26.529907 5003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577196946045-17013-0/minicluster-data/master-0/wal/instance:
uuid: "198ef733e82e4f97be5b8a09af844e3e"
format_stamp: "Formatted at 2026-09-28 06:37:26 on dist-test-slave-rrj5"
server_key: "d5d8b85b54391568e48c2d22346fe965"
server_key_iv: "115c857759d82c153bd947a81ab35a88"
server_key_version: "encryptionkey@0"
I20260928 06:37:26.531103 5003 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:37:26.531747 5018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:26.531909 5003 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:37:26.532006 5003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577196946045-17013-0/minicluster-data/master-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577196946045-17013-0/minicluster-data/master-0/wal
uuid: "198ef733e82e4f97be5b8a09af844e3e"
format_stamp: "Formatted at 2026-09-28 06:37:26 on dist-test-slave-rrj5"
server_key: "d5d8b85b54391568e48c2d22346fe965"
server_key_iv: "115c857759d82c153bd947a81ab35a88"
server_key_version: "encryptionkey@0"
I20260928 06:37:26.532091 5003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577196946045-17013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577196946045-17013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577196946045-17013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:26.535321 5003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:26.536996 5003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:26.537153 5003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:26.540938 5003 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35551
I20260928 06:37:26.541008 5070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35551 every 8 connection(s)
I20260928 06:37:26.541328 5003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
I20260928 06:37:26.542147 5071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:37:26.544123 5071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 198ef733e82e4f97be5b8a09af844e3e: Bootstrap starting.
I20260928 06:37:26.544632 5071 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 198ef733e82e4f97be5b8a09af844e3e: Neither blocks nor log segments found. Creating new log.
I20260928 06:37:26.544864 5071 log.cc:824] T 00000000000000000000000000000000 P 198ef733e82e4f97be5b8a09af844e3e: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:26.545557 5071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 198ef733e82e4f97be5b8a09af844e3e: No bootstrap required, opened a new log
I20260928 06:37:26.547097 5071 raft_consensus.cc:348] T 00000000000000000000000000000000 P 198ef733e82e4f97be5b8a09af844e3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "198ef733e82e4f97be5b8a09af844e3e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35551 } }
I20260928 06:37:26.547228 5071 raft_consensus.cc:374] T 00000000000000000000000000000000 P 198ef733e82e4f97be5b8a09af844e3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:37:26.547250 5071 raft_consensus.cc:729] T 00000000000000000000000000000000 P 198ef733e82e4f97be5b8a09af844e3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 198ef733e82e4f97be5b8a09af844e3e, State: Initialized, Role: FOLLOWER
I20260928 06:37:26.547338 5071 consensus_queue.cc:260] T 00000000000000000000000000000000 P 198ef733e82e4f97be5b8a09af844e3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "198ef733e82e4f97be5b8a09af844e3e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35551 } }
I20260928 06:37:26.547384 5071 raft_consensus.cc:388] T 00000000000000000000000000000000 P 198ef733e82e4f97be5b8a09af844e3e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:37:26.547403 5071 raft_consensus.cc:482] T 00000000000000000000000000000000 P 198ef733e82e4f97be5b8a09af844e3e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:37:26.547431 5071 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 198ef733e82e4f97be5b8a09af844e3e [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:37:26.547870 5071 raft_consensus.cc:504] T 00000000000000000000000000000000 P 198ef733e82e4f97be5b8a09af844e3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "198ef733e82e4f97be5b8a09af844e3e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35551 } }
I20260928 06:37:26.547947 5071 leader_election.cc:302] T 00000000000000000000000000000000 P 198ef733e82e4f97be5b8a09af844e3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 198ef733e82e4f97be5b8a09af844e3e; no voters:
I20260928 06:37:26.548065 5071 leader_election.cc:288] T 00000000000000000000000000000000 P 198ef733e82e4f97be5b8a09af844e3e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:37:26.548123 5074 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 198ef733e82e4f97be5b8a09af844e3e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:37:26.548296 5074 raft_consensus.cc:686] T 00000000000000000000000000000000 P 198ef733e82e4f97be5b8a09af844e3e [term 1 LEADER]: Becoming Leader. State: Replica: 198ef733e82e4f97be5b8a09af844e3e, State: Running, Role: LEADER
I20260928 06:37:26.548327 5071 sys_catalog.cc:564] T 00000000000000000000000000000000 P 198ef733e82e4f97be5b8a09af844e3e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:37:26.548378 5074 consensus_queue.cc:237] T 00000000000000000000000000000000 P 198ef733e82e4f97be5b8a09af844e3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "198ef733e82e4f97be5b8a09af844e3e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35551 } }
I20260928 06:37:26.548763 5076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 198ef733e82e4f97be5b8a09af844e3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 198ef733e82e4f97be5b8a09af844e3e. Latest consensus state: current_term: 1 leader_uuid: "198ef733e82e4f97be5b8a09af844e3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "198ef733e82e4f97be5b8a09af844e3e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35551 } } }
I20260928 06:37:26.548848 5076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 198ef733e82e4f97be5b8a09af844e3e [sys.catalog]: This master's current role is: LEADER
I20260928 06:37:26.548870 5075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 198ef733e82e4f97be5b8a09af844e3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "198ef733e82e4f97be5b8a09af844e3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "198ef733e82e4f97be5b8a09af844e3e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35551 } } }
I20260928 06:37:26.548928 5075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 198ef733e82e4f97be5b8a09af844e3e [sys.catalog]: This master's current role is: LEADER
I20260928 06:37:26.549095 5082 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:37:26.549685 5082 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:37:26.551213 5082 catalog_manager.cc:1347] Generated new cluster ID: 58988f3e5fc343c794521c66158d3c1c
I20260928 06:37:26.551275 5082 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:37:26.551332 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 5003
I20260928 06:37:26.551440 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577196946045-17013-0/minicluster-data/master-0/wal/instance
I20260928 06:37:26.551638 17013 external_mini_cluster.cc:1428] Setting key fff292717e133f42cea607081e45c34f
I20260928 06:37:26.553654 5094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:26.552052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58041 (local address 127.0.0.1:35551)
0928 06:37:26.552229 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:26.552237 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:26.552305 (+ 68us) server_negotiation.cc:407] Connection header received
0928 06:37:26.552456 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:26.552462 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:26.552490 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:26.552611 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:26.553221 (+ 610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:26.553224 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:26.553224 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:37:26.553229 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:26.553267 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:37:26.553280 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:26.553340 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:26.553422 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 06:37:26.553459 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":81,"threads_started":1}
I20260928 06:37:26.554498 17013 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:37:26.557068 5094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:26.555983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41763 (local address 127.0.0.1:35551)
0928 06:37:26.556017 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:26.556019 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:26.556187 (+ 168us) server_negotiation.cc:407] Connection header received
0928 06:37:26.556232 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:26.556233 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:26.556238 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:26.556305 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:26.556868 (+ 563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:26.556869 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:26.556869 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:37:26.556873 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:26.556886 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:37:26.556891 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:26.556952 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:26.556999 (+ 47us) server_negotiation.cc:299] Negotiation successful
0928 06:37:26.557011 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:37:26.576761 5082 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:37:26.576951 5082 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:37:26.585354 5082 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 198ef733e82e4f97be5b8a09af844e3e: Generated new TSK 0
I20260928 06:37:26.585542 5082 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:37:26.593798 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 5003
2026-09-28T06:37:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1313 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577196946045-17013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:37:26 dist-test-slave-rrj5 krb5kdc[5102](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:37:26 dist-test-slave-rrj5 krb5kdc[5102](info): set up 2 sockets
Sep 28 06:37:26 dist-test-slave-rrj5 krb5kdc[5102](info): commencing operation
krb5kdc: starting...
W20260928 06:37:27.627559 17013 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Sep 28 06:37:27 dist-test-slave-rrj5 krb5kdc[5102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:37:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:37:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577196946045-17013-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577196946045-17013-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:37:27.750581 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577196946045-17013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577196946045-17013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577196946045-17013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39223
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577196946045-17013-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39223
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:37:27.830031 5115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:27.830250 5115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:27.830288 5115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:27.831938 5115 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:37:27.832012 5115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:27.832034 5115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:27.832051 5115 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:37:27.832072 5115 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:37:27.834220 5115 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577196946045-17013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577196946045-17013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39223
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577196946045-17013-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577196946045-17013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:37:27.834444 5115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:27.834666 5115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:27.839366 5121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:27.839429 5123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:27.839448 5115 server_base.cc:1034] running on GCE node
W20260928 06:37:27.839355 5120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:27.839797 5115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:27.840016 5115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:27.841197 5115 hybrid_clock.cc:648] HybridClock initialized: now 1790577447841180 us; error 31 us; skew 500 ppm
Sep 28 06:37:27 dist-test-slave-rrj5 krb5kdc[5102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577447, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:27.843552 5115 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:37:27.843950 5115 webserver.cc:460] Webserver started at http://127.0.0.1:37433/ using document root <none> and password file <none>
I20260928 06:37:27.844170 5115 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:27.844237 5115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:27.844393 5115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:37:27.845399 5115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577196946045-17013-0/minicluster-data/master-0/data/instance:
uuid: "4636ee82feb94cd292503f02eebfecbf"
format_stamp: "Formatted at 2026-09-28 06:37:27 on dist-test-slave-rrj5"
server_key: "3da104f316e5c4e27baaa51b877bffa4"
server_key_iv: "ab55d734150a2a9eb03dd5249dbfa037"
server_key_version: "encryptionkey@0"
I20260928 06:37:27.845767 5115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577196946045-17013-0/minicluster-data/master-0/wal/instance:
uuid: "4636ee82feb94cd292503f02eebfecbf"
format_stamp: "Formatted at 2026-09-28 06:37:27 on dist-test-slave-rrj5"
server_key: "3da104f316e5c4e27baaa51b877bffa4"
server_key_iv: "ab55d734150a2a9eb03dd5249dbfa037"
server_key_version: "encryptionkey@0"
I20260928 06:37:27.847029 5115 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:37:27.847692 5130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:27.847851 5115 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:37:27.847944 5115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577196946045-17013-0/minicluster-data/master-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577196946045-17013-0/minicluster-data/master-0/wal
uuid: "4636ee82feb94cd292503f02eebfecbf"
format_stamp: "Formatted at 2026-09-28 06:37:27 on dist-test-slave-rrj5"
server_key: "3da104f316e5c4e27baaa51b877bffa4"
server_key_iv: "ab55d734150a2a9eb03dd5249dbfa037"
server_key_version: "encryptionkey@0"
I20260928 06:37:27.848022 5115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577196946045-17013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577196946045-17013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577196946045-17013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:27.864897 5115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:27.866684 5115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:27.866843 5115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:27.870833 5115 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39223
I20260928 06:37:27.870865 5182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39223 every 8 connection(s)
I20260928 06:37:27.871201 5115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
I20260928 06:37:27.872005 5183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:37:27.874272 5183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4636ee82feb94cd292503f02eebfecbf: Bootstrap starting.
I20260928 06:37:27.874814 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 5115
I20260928 06:37:27.874912 5183 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4636ee82feb94cd292503f02eebfecbf: Neither blocks nor log segments found. Creating new log.
I20260928 06:37:27.874953 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577196946045-17013-0/minicluster-data/master-0/wal/instance
I20260928 06:37:27.875186 17013 external_mini_cluster.cc:1428] Setting key 178b2ed93ccfeec851808f31ad51d58e
I20260928 06:37:27.875252 5183 log.cc:824] T 00000000000000000000000000000000 P 4636ee82feb94cd292503f02eebfecbf: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:27.876183 5183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4636ee82feb94cd292503f02eebfecbf: No bootstrap required, opened a new log
I20260928 06:37:27.878635 5183 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4636ee82feb94cd292503f02eebfecbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4636ee82feb94cd292503f02eebfecbf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39223 } }
I20260928 06:37:27.878809 5183 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4636ee82feb94cd292503f02eebfecbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:37:27.878842 5183 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4636ee82feb94cd292503f02eebfecbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4636ee82feb94cd292503f02eebfecbf, State: Initialized, Role: FOLLOWER
I20260928 06:37:27.879009 5183 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4636ee82feb94cd292503f02eebfecbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4636ee82feb94cd292503f02eebfecbf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39223 } }
I20260928 06:37:27.879079 5183 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4636ee82feb94cd292503f02eebfecbf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:37:27.879101 5183 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4636ee82feb94cd292503f02eebfecbf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:37:27.879172 5183 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4636ee82feb94cd292503f02eebfecbf [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:37:27.879732 5183 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4636ee82feb94cd292503f02eebfecbf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4636ee82feb94cd292503f02eebfecbf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39223 } }
I20260928 06:37:27.879837 5183 leader_election.cc:302] T 00000000000000000000000000000000 P 4636ee82feb94cd292503f02eebfecbf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4636ee82feb94cd292503f02eebfecbf; no voters:
I20260928 06:37:27.879989 5183 leader_election.cc:288] T 00000000000000000000000000000000 P 4636ee82feb94cd292503f02eebfecbf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:37:27.880048 5188 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4636ee82feb94cd292503f02eebfecbf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:37:27.880187 5188 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4636ee82feb94cd292503f02eebfecbf [term 1 LEADER]: Becoming Leader. State: Replica: 4636ee82feb94cd292503f02eebfecbf, State: Running, Role: LEADER
I20260928 06:37:27.880215 5183 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4636ee82feb94cd292503f02eebfecbf [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:37:27.880326 5188 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4636ee82feb94cd292503f02eebfecbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4636ee82feb94cd292503f02eebfecbf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39223 } }
I20260928 06:37:27.880630 5188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4636ee82feb94cd292503f02eebfecbf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4636ee82feb94cd292503f02eebfecbf. Latest consensus state: current_term: 1 leader_uuid: "4636ee82feb94cd292503f02eebfecbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4636ee82feb94cd292503f02eebfecbf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39223 } } }
I20260928 06:37:27.880775 5188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4636ee82feb94cd292503f02eebfecbf [sys.catalog]: This master's current role is: LEADER
I20260928 06:37:27.880775 5190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4636ee82feb94cd292503f02eebfecbf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4636ee82feb94cd292503f02eebfecbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4636ee82feb94cd292503f02eebfecbf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39223 } } }
I20260928 06:37:27.880957 5190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4636ee82feb94cd292503f02eebfecbf [sys.catalog]: This master's current role is: LEADER
I20260928 06:37:27.881114 5197 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:37:27.881880 5197 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 06:37:27 dist-test-slave-rrj5 krb5kdc[5102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 06:37:27.883536 5197 catalog_manager.cc:1347] Generated new cluster ID: 12ee16395baa42ca96c92d7828358159
I20260928 06:37:27.883589 5197 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:37:27.886929 5187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:27.875718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54263 (local address 127.0.0.1:39223)
0928 06:37:27.875890 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:27.875899 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:27.875940 (+ 41us) server_negotiation.cc:407] Connection header received
0928 06:37:27.876546 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:27.876555 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:27.876616 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:27.876771 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:27.877355 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:27.878837 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:27.879701 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:27.880347 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:27.880507 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:27.883434 (+ 2927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:27.883438 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:27.883438 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:27.883449 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:27.885206 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:27.885606 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:27.885609 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:27.885609 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:27.885701 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:27.885916 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:27.885917 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:27.885917 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:27.886150 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:27.886325 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:27.886435 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 06:37:27.886760 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":97,"threads_started":1}
I20260928 06:37:27.887804 17013 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:37:27.892593 5197 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:37:27.893225 5197 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:37:27.895347 5187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:27.889094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58081 (local address 127.0.0.1:39223)
0928 06:37:27.889130 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:27.889132 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:27.889260 (+ 128us) server_negotiation.cc:407] Connection header received
0928 06:37:27.889322 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:27.889323 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:27.889332 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:27.889380 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:27.889979 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:27.890454 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:27.891145 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:27.891377 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:27.891449 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:27.892756 (+ 1307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:27.892757 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:27.892758 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:27.892764 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:27.894221 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:27.894585 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:27.894586 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:27.894586 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:27.894638 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:27.894840 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:27.894841 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:27.894841 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:27.894952 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:27.895049 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:27.895162 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 06:37:27.895243 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":20}
I20260928 06:37:27.907272 5197 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4636ee82feb94cd292503f02eebfecbf: Generated new TSK 0
I20260928 06:37:27.907469 5197 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:37:27.909319 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 5115
2026-09-28T06:37:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1315 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577196946045-17013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:37:27 dist-test-slave-rrj5 krb5kdc[5214](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:37:27 dist-test-slave-rrj5 krb5kdc[5214](info): set up 2 sockets
Sep 28 06:37:27 dist-test-slave-rrj5 krb5kdc[5214](info): commencing operation
krb5kdc: starting...
W20260928 06:37:28.958493 17013 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Sep 28 06:37:29 dist-test-slave-rrj5 krb5kdc[5214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577449, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:37:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:37:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577196946045-17013-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577196946045-17013-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:37:29.072930 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577196946045-17013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577196946045-17013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577196946045-17013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35785
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577196946045-17013-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35785
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:37:29.151995 5227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:29.152240 5227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:29.152276 5227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:29.153416 5227 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:37:29.153491 5227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:29.153515 5227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:29.153544 5227 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:37:29.153606 5227 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:37:29.155098 5227 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577196946045-17013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577196946045-17013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35785
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577196946045-17013-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577196946045-17013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:37:29.155368 5227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:29.155607 5227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:29.160738 5233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:29.160696 5232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:29.160694 5235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:29.161132 5227 server_base.cc:1034] running on GCE node
I20260928 06:37:29.161362 5227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:29.161672 5227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:29.162856 5227 hybrid_clock.cc:648] HybridClock initialized: now 1790577449162841 us; error 35 us; skew 500 ppm
Sep 28 06:37:29 dist-test-slave-rrj5 krb5kdc[5214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577449, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:29.165167 5227 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:37:29.165632 5227 webserver.cc:460] Webserver started at http://127.0.0.1:33819/ using document root <none> and password file <none>
I20260928 06:37:29.165856 5227 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:29.165921 5227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:29.166074 5227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:37:29.167166 5227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577196946045-17013-0/minicluster-data/master-0/data/instance:
uuid: "ce8bb97047ea46be95730265518860dc"
format_stamp: "Formatted at 2026-09-28 06:37:29 on dist-test-slave-rrj5"
server_key: "31c5592ee88cc5fad22d28247b4f4f1b"
server_key_iv: "1329862ba57e93c1e2fb8895b2c67b1f"
server_key_version: "encryptionkey@0"
I20260928 06:37:29.167515 5227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577196946045-17013-0/minicluster-data/master-0/wal/instance:
uuid: "ce8bb97047ea46be95730265518860dc"
format_stamp: "Formatted at 2026-09-28 06:37:29 on dist-test-slave-rrj5"
server_key: "31c5592ee88cc5fad22d28247b4f4f1b"
server_key_iv: "1329862ba57e93c1e2fb8895b2c67b1f"
server_key_version: "encryptionkey@0"
I20260928 06:37:29.168826 5227 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:37:29.169523 5242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:29.169710 5227 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:37:29.169792 5227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577196946045-17013-0/minicluster-data/master-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577196946045-17013-0/minicluster-data/master-0/wal
uuid: "ce8bb97047ea46be95730265518860dc"
format_stamp: "Formatted at 2026-09-28 06:37:29 on dist-test-slave-rrj5"
server_key: "31c5592ee88cc5fad22d28247b4f4f1b"
server_key_iv: "1329862ba57e93c1e2fb8895b2c67b1f"
server_key_version: "encryptionkey@0"
I20260928 06:37:29.169844 5227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577196946045-17013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577196946045-17013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577196946045-17013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:29.178297 5227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:29.180500 5227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:29.180613 5227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:29.184405 5227 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35785
I20260928 06:37:29.184438 5294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35785 every 8 connection(s)
I20260928 06:37:29.184772 5227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
I20260928 06:37:29.185667 5295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:37:29.187160 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 5227
I20260928 06:37:29.187296 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577196946045-17013-0/minicluster-data/master-0/wal/instance
I20260928 06:37:29.187484 17013 external_mini_cluster.cc:1428] Setting key 1bef7304c2a6efd0f807020e51656531
I20260928 06:37:29.188090 5295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce8bb97047ea46be95730265518860dc: Bootstrap starting.
I20260928 06:37:29.188683 5295 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce8bb97047ea46be95730265518860dc: Neither blocks nor log segments found. Creating new log.
I20260928 06:37:29.189030 5295 log.cc:824] T 00000000000000000000000000000000 P ce8bb97047ea46be95730265518860dc: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:29.189767 5295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce8bb97047ea46be95730265518860dc: No bootstrap required, opened a new log
I20260928 06:37:29.192031 5295 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce8bb97047ea46be95730265518860dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce8bb97047ea46be95730265518860dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35785 } }
I20260928 06:37:29.192219 5295 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce8bb97047ea46be95730265518860dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:37:29.192269 5295 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce8bb97047ea46be95730265518860dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce8bb97047ea46be95730265518860dc, State: Initialized, Role: FOLLOWER
I20260928 06:37:29.192408 5295 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce8bb97047ea46be95730265518860dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce8bb97047ea46be95730265518860dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35785 } }
I20260928 06:37:29.192510 5295 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce8bb97047ea46be95730265518860dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:37:29.192567 5295 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce8bb97047ea46be95730265518860dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:37:29.192617 5295 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce8bb97047ea46be95730265518860dc [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:37:29.193189 5298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:29.187963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33851 (local address 127.0.0.1:35785)
0928 06:37:29.188146 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:29.188180 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:29.188262 (+ 82us) server_negotiation.cc:407] Connection header received
0928 06:37:29.188747 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:29.188755 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:29.188826 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:29.188934 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:29.189188 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:29.190248 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:29.191349 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:29.191742 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:29.191819 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:29.192488 (+ 669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:29.192494 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:29.192494 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:37:29.192714 (+ 220us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:29.192754 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:37:29.192766 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:29.192816 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:29.192917 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 06:37:29.192985 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":92,"threads_started":1}
I20260928 06:37:29.193257 5295 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce8bb97047ea46be95730265518860dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce8bb97047ea46be95730265518860dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35785 } }
I20260928 06:37:29.193387 5295 leader_election.cc:302] T 00000000000000000000000000000000 P ce8bb97047ea46be95730265518860dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce8bb97047ea46be95730265518860dc; no voters:
I20260928 06:37:29.193542 5295 leader_election.cc:288] T 00000000000000000000000000000000 P ce8bb97047ea46be95730265518860dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:37:29.193813 5295 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce8bb97047ea46be95730265518860dc [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:37:29.193837 5301 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce8bb97047ea46be95730265518860dc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:37:29.193954 5301 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce8bb97047ea46be95730265518860dc [term 1 LEADER]: Becoming Leader. State: Replica: ce8bb97047ea46be95730265518860dc, State: Running, Role: LEADER
I20260928 06:37:29.194095 5301 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce8bb97047ea46be95730265518860dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce8bb97047ea46be95730265518860dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35785 } }
I20260928 06:37:29.194156 17013 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:37:29.194510 5301 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce8bb97047ea46be95730265518860dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce8bb97047ea46be95730265518860dc. Latest consensus state: current_term: 1 leader_uuid: "ce8bb97047ea46be95730265518860dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce8bb97047ea46be95730265518860dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35785 } } }
I20260928 06:37:29.194651 5301 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce8bb97047ea46be95730265518860dc [sys.catalog]: This master's current role is: LEADER
I20260928 06:37:29.194944 5309 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:37:29.195600 5309 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:37:29.195670 5302 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce8bb97047ea46be95730265518860dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce8bb97047ea46be95730265518860dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce8bb97047ea46be95730265518860dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35785 } } }
I20260928 06:37:29.195786 5302 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce8bb97047ea46be95730265518860dc [sys.catalog]: This master's current role is: LEADER
I20260928 06:37:29.197690 5309 catalog_manager.cc:1347] Generated new cluster ID: 504b69af91b5402e90684ab7eea46abd
I20260928 06:37:29.197739 5309 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:37:29.199129 5298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:29.195827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53181 (local address 127.0.0.1:35785)
0928 06:37:29.195860 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:29.195862 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:29.195998 (+ 136us) server_negotiation.cc:407] Connection header received
0928 06:37:29.196062 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:29.196062 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:29.196072 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:29.196139 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:29.196288 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:29.196748 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:29.197764 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:29.198091 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:29.198147 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:29.198868 (+ 721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:29.198870 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:29.198870 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:37:29.198875 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:29.198893 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:37:29.198898 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:29.198927 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:29.199038 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 06:37:29.199060 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 06:37:29.206427 5309 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:37:29.206925 5309 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:37:29.214437 5309 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce8bb97047ea46be95730265518860dc: Generated new TSK 0
I20260928 06:37:29.214656 5309 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:37:29.220160 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 5227
2026-09-28T06:37:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1311 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577196946045-17013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:37:29 dist-test-slave-rrj5 krb5kdc[5326](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:37:29 dist-test-slave-rrj5 krb5kdc[5326](info): set up 2 sockets
Sep 28 06:37:29 dist-test-slave-rrj5 krb5kdc[5326](info): commencing operation
krb5kdc: starting...
W20260928 06:37:30.238502 17013 mini_kdc.cc:121] Time spent starting KDC: real 1.000s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Sep 28 06:37:30 dist-test-slave-rrj5 krb5kdc[5326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:37:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:37:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577196946045-17013-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577196946045-17013-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:37:30.356456 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577196946045-17013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577196946045-17013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577196946045-17013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44069
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577196946045-17013-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44069
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:37:30.437166 5339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:30.437422 5339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:30.437463 5339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:30.438736 5339 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:37:30.438808 5339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:30.438833 5339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:30.438862 5339 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:37:30.438895 5339 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:37:30.440403 5339 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577196946045-17013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577196946045-17013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44069
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577196946045-17013-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5339
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577196946045-17013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:37:30.440694 5339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:30.440932 5339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:30.445879 5344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:30.445870 5347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:30.445870 5345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:30.446223 5339 server_base.cc:1034] running on GCE node
I20260928 06:37:30.446493 5339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:30.446789 5339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:30.447935 5339 hybrid_clock.cc:648] HybridClock initialized: now 1790577450447915 us; error 35 us; skew 500 ppm
Sep 28 06:37:30 dist-test-slave-rrj5 krb5kdc[5326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577450, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:30.455322 5339 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:37:30.455847 5339 webserver.cc:460] Webserver started at http://127.0.0.1:41347/ using document root <none> and password file <none>
I20260928 06:37:30.456079 5339 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:30.456132 5339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:30.456310 5339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:37:30.457387 5339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577196946045-17013-0/minicluster-data/master-0/data/instance:
uuid: "8888bf4ef1424c5eac92943e3ae7c796"
format_stamp: "Formatted at 2026-09-28 06:37:30 on dist-test-slave-rrj5"
server_key: "71ebebb28dc1cae73544bbf2b46cef5d"
server_key_iv: "52121762168f0606240321dfa9ecf745"
server_key_version: "encryptionkey@0"
I20260928 06:37:30.457790 5339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577196946045-17013-0/minicluster-data/master-0/wal/instance:
uuid: "8888bf4ef1424c5eac92943e3ae7c796"
format_stamp: "Formatted at 2026-09-28 06:37:30 on dist-test-slave-rrj5"
server_key: "71ebebb28dc1cae73544bbf2b46cef5d"
server_key_iv: "52121762168f0606240321dfa9ecf745"
server_key_version: "encryptionkey@0"
I20260928 06:37:30.459164 5339 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:37:30.459895 5354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:30.460067 5339 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:37:30.460182 5339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577196946045-17013-0/minicluster-data/master-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577196946045-17013-0/minicluster-data/master-0/wal
uuid: "8888bf4ef1424c5eac92943e3ae7c796"
format_stamp: "Formatted at 2026-09-28 06:37:30 on dist-test-slave-rrj5"
server_key: "71ebebb28dc1cae73544bbf2b46cef5d"
server_key_iv: "52121762168f0606240321dfa9ecf745"
server_key_version: "encryptionkey@0"
I20260928 06:37:30.460258 5339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577196946045-17013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577196946045-17013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577196946045-17013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:30.463716 5339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:30.465556 5339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:30.465780 5339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:30.469856 5339 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44069
I20260928 06:37:30.469885 5406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44069 every 8 connection(s)
I20260928 06:37:30.470237 5339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
I20260928 06:37:30.471072 5407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:37:30.471607 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 5339
I20260928 06:37:30.471720 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577196946045-17013-0/minicluster-data/master-0/wal/instance
I20260928 06:37:30.471941 17013 external_mini_cluster.cc:1428] Setting key 5bc1c198a7ebe0cd1f6e91d89e46c577
I20260928 06:37:30.474270 5407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8888bf4ef1424c5eac92943e3ae7c796: Bootstrap starting.
I20260928 06:37:30.474499 5410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:30.472501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43291 (local address 127.0.0.1:44069)
0928 06:37:30.472675 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:30.472691 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:30.472716 (+ 25us) server_negotiation.cc:407] Connection header received
0928 06:37:30.473098 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:30.473104 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:30.473131 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:30.473254 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:30.473987 (+ 733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:30.473991 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:30.473991 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:37:30.473994 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:30.474032 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:37:30.474042 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:30.474099 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:30.474198 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 06:37:30.474254 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":80,"threads_started":1}
I20260928 06:37:30.475040 5407 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8888bf4ef1424c5eac92943e3ae7c796: Neither blocks nor log segments found. Creating new log.
I20260928 06:37:30.475400 5407 log.cc:824] T 00000000000000000000000000000000 P 8888bf4ef1424c5eac92943e3ae7c796: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:30.475512 17013 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:37:30.476181 5407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8888bf4ef1424c5eac92943e3ae7c796: No bootstrap required, opened a new log
I20260928 06:37:30.478202 5410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:30.477098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36729 (local address 127.0.0.1:44069)
0928 06:37:30.477139 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:30.477141 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:30.477258 (+ 117us) server_negotiation.cc:407] Connection header received
0928 06:37:30.477315 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:30.477315 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:30.477319 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:30.477365 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:30.477971 (+ 606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:30.477972 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:30.477972 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:37:30.477976 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:30.477988 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:37:30.477992 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:30.478031 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:30.478104 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 06:37:30.478114 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 06:37:30.478930 5407 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8888bf4ef1424c5eac92943e3ae7c796 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8888bf4ef1424c5eac92943e3ae7c796" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44069 } }
I20260928 06:37:30.479064 5407 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8888bf4ef1424c5eac92943e3ae7c796 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:37:30.479087 5407 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8888bf4ef1424c5eac92943e3ae7c796 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8888bf4ef1424c5eac92943e3ae7c796, State: Initialized, Role: FOLLOWER
I20260928 06:37:30.479203 5407 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8888bf4ef1424c5eac92943e3ae7c796 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8888bf4ef1424c5eac92943e3ae7c796" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44069 } }
I20260928 06:37:30.479295 5407 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8888bf4ef1424c5eac92943e3ae7c796 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:37:30.479317 5407 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8888bf4ef1424c5eac92943e3ae7c796 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:37:30.479346 5407 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8888bf4ef1424c5eac92943e3ae7c796 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:37:30.479918 5407 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8888bf4ef1424c5eac92943e3ae7c796 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8888bf4ef1424c5eac92943e3ae7c796" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44069 } }
I20260928 06:37:30.480005 5407 leader_election.cc:302] T 00000000000000000000000000000000 P 8888bf4ef1424c5eac92943e3ae7c796 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8888bf4ef1424c5eac92943e3ae7c796; no voters:
I20260928 06:37:30.480129 5407 leader_election.cc:288] T 00000000000000000000000000000000 P 8888bf4ef1424c5eac92943e3ae7c796 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:37:30.480212 5417 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8888bf4ef1424c5eac92943e3ae7c796 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:37:30.480362 5417 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8888bf4ef1424c5eac92943e3ae7c796 [term 1 LEADER]: Becoming Leader. State: Replica: 8888bf4ef1424c5eac92943e3ae7c796, State: Running, Role: LEADER
I20260928 06:37:30.480361 5407 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8888bf4ef1424c5eac92943e3ae7c796 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:37:30.480479 5417 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8888bf4ef1424c5eac92943e3ae7c796 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8888bf4ef1424c5eac92943e3ae7c796" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44069 } }
I20260928 06:37:30.480930 5418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8888bf4ef1424c5eac92943e3ae7c796 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8888bf4ef1424c5eac92943e3ae7c796" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8888bf4ef1424c5eac92943e3ae7c796" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44069 } } }
I20260928 06:37:30.480938 5419 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8888bf4ef1424c5eac92943e3ae7c796 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8888bf4ef1424c5eac92943e3ae7c796. Latest consensus state: current_term: 1 leader_uuid: "8888bf4ef1424c5eac92943e3ae7c796" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8888bf4ef1424c5eac92943e3ae7c796" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44069 } } }
I20260928 06:37:30.481138 5418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8888bf4ef1424c5eac92943e3ae7c796 [sys.catalog]: This master's current role is: LEADER
I20260928 06:37:30.481153 5419 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8888bf4ef1424c5eac92943e3ae7c796 [sys.catalog]: This master's current role is: LEADER
I20260928 06:37:30.481395 5426 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:37:30.481942 5426 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:37:30.483572 5426 catalog_manager.cc:1347] Generated new cluster ID: b7142ebcbfdd411e90c2c96466a08e31
I20260928 06:37:30.483649 5426 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:37:30.498127 5426 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:37:30.498323 5426 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:37:30.502992 5426 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8888bf4ef1424c5eac92943e3ae7c796: Generated new TSK 0
I20260928 06:37:30.503201 5426 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:37:30.507159 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 5339
2026-09-28T06:37:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1287 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577196946045-17013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:37:30 dist-test-slave-rrj5 krb5kdc[5438](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:37:30 dist-test-slave-rrj5 krb5kdc[5438](info): set up 2 sockets
Sep 28 06:37:30 dist-test-slave-rrj5 krb5kdc[5438](info): commencing operation
krb5kdc: starting...
W20260928 06:37:31.538446 17013 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Sep 28 06:37:31 dist-test-slave-rrj5 krb5kdc[5438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577451, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:37:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:37:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577196946045-17013-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577196946045-17013-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:37:31.655838 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577196946045-17013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577196946045-17013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577196946045-17013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41909
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577196946045-17013-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41909
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:37:31.734992 5451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:31.735258 5451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:31.735301 5451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:31.736512 5451 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:37:31.736588 5451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:31.736613 5451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:31.736634 5451 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:37:31.736662 5451 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:37:31.738339 5451 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577196946045-17013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577196946045-17013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41909
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577196946045-17013-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577196946045-17013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:37:31.738612 5451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:31.738911 5451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:31.743806 5456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:31.743772 5459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:31.743785 5457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:31.743820 5451 server_base.cc:1034] running on GCE node
I20260928 06:37:31.744266 5451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:31.744551 5451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:31.745649 5451 hybrid_clock.cc:648] HybridClock initialized: now 1790577451745631 us; error 39 us; skew 500 ppm
Sep 28 06:37:31 dist-test-slave-rrj5 krb5kdc[5438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577451, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:31.747929 5451 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:37:31.748420 5451 webserver.cc:460] Webserver started at http://127.0.0.1:45863/ using document root <none> and password file <none>
I20260928 06:37:31.748644 5451 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:31.748692 5451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:31.748842 5451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:37:31.749974 5451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577196946045-17013-0/minicluster-data/master-0/data/instance:
uuid: "7f6acb080abd461194f60f0e052eb846"
format_stamp: "Formatted at 2026-09-28 06:37:31 on dist-test-slave-rrj5"
server_key: "810536d8fe77b0902b93f96cfda23d4b"
server_key_iv: "c89223d983168001319afa05de716bc6"
server_key_version: "encryptionkey@0"
I20260928 06:37:31.750335 5451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577196946045-17013-0/minicluster-data/master-0/wal/instance:
uuid: "7f6acb080abd461194f60f0e052eb846"
format_stamp: "Formatted at 2026-09-28 06:37:31 on dist-test-slave-rrj5"
server_key: "810536d8fe77b0902b93f96cfda23d4b"
server_key_iv: "c89223d983168001319afa05de716bc6"
server_key_version: "encryptionkey@0"
I20260928 06:37:31.751761 5451 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:37:31.752523 5466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:31.752725 5451 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:37:31.752801 5451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577196946045-17013-0/minicluster-data/master-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577196946045-17013-0/minicluster-data/master-0/wal
uuid: "7f6acb080abd461194f60f0e052eb846"
format_stamp: "Formatted at 2026-09-28 06:37:31 on dist-test-slave-rrj5"
server_key: "810536d8fe77b0902b93f96cfda23d4b"
server_key_iv: "c89223d983168001319afa05de716bc6"
server_key_version: "encryptionkey@0"
I20260928 06:37:31.752884 5451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577196946045-17013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577196946045-17013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577196946045-17013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:31.780671 5451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:31.782691 5451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:31.782836 5451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:31.786888 5451 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41909
I20260928 06:37:31.786927 5518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41909 every 8 connection(s)
I20260928 06:37:31.787292 5451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
I20260928 06:37:31.788125 5519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:37:31.790278 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 5451
I20260928 06:37:31.790416 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577196946045-17013-0/minicluster-data/master-0/wal/instance
I20260928 06:37:31.790583 5519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f6acb080abd461194f60f0e052eb846: Bootstrap starting.
I20260928 06:37:31.790627 17013 external_mini_cluster.cc:1428] Setting key ab2f1cf2d45d9aba01b9d346d7881761
I20260928 06:37:31.791352 5519 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f6acb080abd461194f60f0e052eb846: Neither blocks nor log segments found. Creating new log.
I20260928 06:37:31.791678 5519 log.cc:824] T 00000000000000000000000000000000 P 7f6acb080abd461194f60f0e052eb846: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:31.792434 5519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f6acb080abd461194f60f0e052eb846: No bootstrap required, opened a new log
I20260928 06:37:31.794790 5519 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f6acb080abd461194f60f0e052eb846 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f6acb080abd461194f60f0e052eb846" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41909 } }
I20260928 06:37:31.794921 5519 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f6acb080abd461194f60f0e052eb846 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:37:31.794950 5519 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f6acb080abd461194f60f0e052eb846 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f6acb080abd461194f60f0e052eb846, State: Initialized, Role: FOLLOWER
I20260928 06:37:31.795085 5519 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f6acb080abd461194f60f0e052eb846 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f6acb080abd461194f60f0e052eb846" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41909 } }
I20260928 06:37:31.795161 5519 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f6acb080abd461194f60f0e052eb846 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:37:31.795202 5519 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f6acb080abd461194f60f0e052eb846 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:37:31.795249 5519 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f6acb080abd461194f60f0e052eb846 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:37:31.795903 5519 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f6acb080abd461194f60f0e052eb846 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f6acb080abd461194f60f0e052eb846" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41909 } }
I20260928 06:37:31.796015 5519 leader_election.cc:302] T 00000000000000000000000000000000 P 7f6acb080abd461194f60f0e052eb846 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f6acb080abd461194f60f0e052eb846; no voters:
I20260928 06:37:31.796180 5519 leader_election.cc:288] T 00000000000000000000000000000000 P 7f6acb080abd461194f60f0e052eb846 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:37:31.796247 5524 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f6acb080abd461194f60f0e052eb846 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:37:31.796862 5524 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f6acb080abd461194f60f0e052eb846 [term 1 LEADER]: Becoming Leader. State: Replica: 7f6acb080abd461194f60f0e052eb846, State: Running, Role: LEADER
I20260928 06:37:31.796980 5519 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f6acb080abd461194f60f0e052eb846 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:37:31.797045 5524 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f6acb080abd461194f60f0e052eb846 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f6acb080abd461194f60f0e052eb846" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41909 } }
I20260928 06:37:31.797536 5525 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f6acb080abd461194f60f0e052eb846 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f6acb080abd461194f60f0e052eb846" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f6acb080abd461194f60f0e052eb846" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41909 } } }
I20260928 06:37:31.797554 5526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f6acb080abd461194f60f0e052eb846 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f6acb080abd461194f60f0e052eb846. Latest consensus state: current_term: 1 leader_uuid: "7f6acb080abd461194f60f0e052eb846" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f6acb080abd461194f60f0e052eb846" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41909 } } }
I20260928 06:37:31.797667 5525 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f6acb080abd461194f60f0e052eb846 [sys.catalog]: This master's current role is: LEADER
I20260928 06:37:31.797688 5526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f6acb080abd461194f60f0e052eb846 [sys.catalog]: This master's current role is: LEADER
I20260928 06:37:31.798002 5533 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 28 06:37:31 dist-test-slave-rrj5 krb5kdc[5438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577451, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 06:37:31.798532 5533 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:37:31.800187 5533 catalog_manager.cc:1347] Generated new cluster ID: 08c6a35937dd4267a433f664b790a4d4
I20260928 06:37:31.800241 5533 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:37:31.803459 5522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:31.791180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.60:48909 (local address 127.0.0.1:41909)
0928 06:37:31.791382 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:31.791392 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:31.791427 (+ 35us) server_negotiation.cc:407] Connection header received
0928 06:37:31.792058 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:31.792065 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:31.792155 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:31.792315 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:31.792969 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:31.794412 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:31.795303 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:31.795915 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:31.796031 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:31.799220 (+ 3189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:31.799223 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:31.799224 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:31.799234 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:31.801365 (+ 2131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:31.801929 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:31.801932 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:31.801932 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:31.802039 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:31.802319 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:31.802319 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:31.802320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:31.802624 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:31.802761 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:31.803092 (+ 331us) server_negotiation.cc:299] Negotiation successful
0928 06:37:31.803209 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":88,"threads_started":1}
I20260928 06:37:31.804894 17013 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:37:31.810422 5533 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:37:31.811033 5533 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:37:31.813222 5522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:31.806518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.60:37685 (local address 127.0.0.1:41909)
0928 06:37:31.806549 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:31.806551 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:31.806584 (+ 33us) server_negotiation.cc:407] Connection header received
0928 06:37:31.806657 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:31.806658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:31.806666 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:31.806714 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:31.807316 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:31.808019 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:31.808970 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:31.809256 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:31.809322 (+ 66us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:31.810577 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:31.810578 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:31.810578 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:31.810584 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:31.812057 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:31.812479 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:31.812480 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:31.812480 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:31.812531 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:31.812752 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:31.812753 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:31.812753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:31.812856 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:31.812952 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:31.813089 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 06:37:31.813144 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 06:37:31.818452 5533 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f6acb080abd461194f60f0e052eb846: Generated new TSK 0
I20260928 06:37:31.818616 5533 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:37:31.822167 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 5451
W20260928 06:37:31.827195 5436 connection.cc:535] client connection to 127.0.0.1:41909 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41909): Connection reset by peer (error 104)
2026-09-28T06:37:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1315 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577196946045-17013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:37:31 dist-test-slave-rrj5 krb5kdc[5550](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:37:31 dist-test-slave-rrj5 krb5kdc[5550](info): set up 2 sockets
Sep 28 06:37:31 dist-test-slave-rrj5 krb5kdc[5550](info): commencing operation
krb5kdc: starting...
W20260928 06:37:32.866576 17013 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Sep 28 06:37:32 dist-test-slave-rrj5 krb5kdc[5550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:37:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:37:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577196946045-17013-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577196946045-17013-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:37:32.984817 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577196946045-17013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577196946045-17013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577196946045-17013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42487
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577196946045-17013-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42487
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:37:33.064483 5563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:33.064699 5563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:33.064723 5563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:33.065876 5563 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:37:33.065912 5563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:33.065927 5563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:33.065938 5563 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:37:33.065951 5563 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:37:33.067339 5563 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577196946045-17013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577196946045-17013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42487
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577196946045-17013-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577196946045-17013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:37:33.067546 5563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:33.067721 5563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:33.072467 5569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:33.072670 5563 server_base.cc:1034] running on GCE node
W20260928 06:37:33.072480 5571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:33.072497 5568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:33.073022 5563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:33.073374 5563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:33.074543 5563 hybrid_clock.cc:648] HybridClock initialized: now 1790577453074521 us; error 34 us; skew 500 ppm
Sep 28 06:37:33 dist-test-slave-rrj5 krb5kdc[5550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:33.077039 5563 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:37:33.077514 5563 webserver.cc:460] Webserver started at http://127.0.0.1:34051/ using document root <none> and password file <none>
I20260928 06:37:33.077780 5563 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:33.077854 5563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:33.078009 5563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:37:33.079177 5563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577196946045-17013-0/minicluster-data/master-0/data/instance:
uuid: "9b0f1dad41ea4ac2ac25af6f105ccdb8"
format_stamp: "Formatted at 2026-09-28 06:37:33 on dist-test-slave-rrj5"
server_key: "84ed2d904e6347b87fd42a2b20a93460"
server_key_iv: "c393f39aeebc6685f71897e4047f60dc"
server_key_version: "encryptionkey@0"
I20260928 06:37:33.079542 5563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577196946045-17013-0/minicluster-data/master-0/wal/instance:
uuid: "9b0f1dad41ea4ac2ac25af6f105ccdb8"
format_stamp: "Formatted at 2026-09-28 06:37:33 on dist-test-slave-rrj5"
server_key: "84ed2d904e6347b87fd42a2b20a93460"
server_key_iv: "c393f39aeebc6685f71897e4047f60dc"
server_key_version: "encryptionkey@0"
I20260928 06:37:33.080868 5563 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:37:33.081631 5578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:33.081801 5563 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:37:33.081904 5563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577196946045-17013-0/minicluster-data/master-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577196946045-17013-0/minicluster-data/master-0/wal
uuid: "9b0f1dad41ea4ac2ac25af6f105ccdb8"
format_stamp: "Formatted at 2026-09-28 06:37:33 on dist-test-slave-rrj5"
server_key: "84ed2d904e6347b87fd42a2b20a93460"
server_key_iv: "c393f39aeebc6685f71897e4047f60dc"
server_key_version: "encryptionkey@0"
I20260928 06:37:33.081987 5563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577196946045-17013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577196946045-17013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577196946045-17013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:33.097931 5563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:33.100392 5563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:33.100562 5563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:33.104702 5563 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42487
I20260928 06:37:33.104740 5630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42487 every 8 connection(s)
I20260928 06:37:33.105110 5563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
I20260928 06:37:33.105965 5631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:37:33.108160 5631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b0f1dad41ea4ac2ac25af6f105ccdb8: Bootstrap starting.
I20260928 06:37:33.108669 5631 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b0f1dad41ea4ac2ac25af6f105ccdb8: Neither blocks nor log segments found. Creating new log.
I20260928 06:37:33.108898 5631 log.cc:824] T 00000000000000000000000000000000 P 9b0f1dad41ea4ac2ac25af6f105ccdb8: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:33.109462 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 5563
I20260928 06:37:33.109547 5631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b0f1dad41ea4ac2ac25af6f105ccdb8: No bootstrap required, opened a new log
I20260928 06:37:33.109624 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577196946045-17013-0/minicluster-data/master-0/wal/instance
I20260928 06:37:33.109805 17013 external_mini_cluster.cc:1428] Setting key aec707ba64496d9255fe00010a831e4a
I20260928 06:37:33.112200 5631 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b0f1dad41ea4ac2ac25af6f105ccdb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b0f1dad41ea4ac2ac25af6f105ccdb8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42487 } }
I20260928 06:37:33.112402 5631 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b0f1dad41ea4ac2ac25af6f105ccdb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:37:33.112439 5631 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b0f1dad41ea4ac2ac25af6f105ccdb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b0f1dad41ea4ac2ac25af6f105ccdb8, State: Initialized, Role: FOLLOWER
I20260928 06:37:33.112562 5631 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b0f1dad41ea4ac2ac25af6f105ccdb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b0f1dad41ea4ac2ac25af6f105ccdb8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42487 } }
I20260928 06:37:33.112635 5631 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b0f1dad41ea4ac2ac25af6f105ccdb8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:37:33.112672 5631 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b0f1dad41ea4ac2ac25af6f105ccdb8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:37:33.112716 5631 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b0f1dad41ea4ac2ac25af6f105ccdb8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:37:33.113363 5631 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b0f1dad41ea4ac2ac25af6f105ccdb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b0f1dad41ea4ac2ac25af6f105ccdb8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42487 } }
I20260928 06:37:33.113484 5631 leader_election.cc:302] T 00000000000000000000000000000000 P 9b0f1dad41ea4ac2ac25af6f105ccdb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b0f1dad41ea4ac2ac25af6f105ccdb8; no voters:
I20260928 06:37:33.113652 5631 leader_election.cc:288] T 00000000000000000000000000000000 P 9b0f1dad41ea4ac2ac25af6f105ccdb8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:37:33.113677 5636 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b0f1dad41ea4ac2ac25af6f105ccdb8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:37:33.113781 5636 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b0f1dad41ea4ac2ac25af6f105ccdb8 [term 1 LEADER]: Becoming Leader. State: Replica: 9b0f1dad41ea4ac2ac25af6f105ccdb8, State: Running, Role: LEADER
I20260928 06:37:33.113907 5631 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b0f1dad41ea4ac2ac25af6f105ccdb8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:37:33.113937 5636 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b0f1dad41ea4ac2ac25af6f105ccdb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b0f1dad41ea4ac2ac25af6f105ccdb8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42487 } }
I20260928 06:37:33.114555 5638 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b0f1dad41ea4ac2ac25af6f105ccdb8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b0f1dad41ea4ac2ac25af6f105ccdb8. Latest consensus state: current_term: 1 leader_uuid: "9b0f1dad41ea4ac2ac25af6f105ccdb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b0f1dad41ea4ac2ac25af6f105ccdb8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42487 } } }
I20260928 06:37:33.114573 5637 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b0f1dad41ea4ac2ac25af6f105ccdb8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b0f1dad41ea4ac2ac25af6f105ccdb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b0f1dad41ea4ac2ac25af6f105ccdb8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42487 } } }
I20260928 06:37:33.114652 5638 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b0f1dad41ea4ac2ac25af6f105ccdb8 [sys.catalog]: This master's current role is: LEADER
I20260928 06:37:33.114674 5637 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b0f1dad41ea4ac2ac25af6f105ccdb8 [sys.catalog]: This master's current role is: LEADER
I20260928 06:37:33.114848 5645 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:37:33.115346 5645 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:37:33.116322 5635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:33.110318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.60:39699 (local address 127.0.0.1:42487)
0928 06:37:33.110482 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:33.110490 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:33.110514 (+ 24us) server_negotiation.cc:407] Connection header received
0928 06:37:33.110909 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:33.110914 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:33.110959 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:33.111066 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:33.111327 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:33.112186 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:33.113298 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:33.113873 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:33.114315 (+ 442us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:33.115425 (+ 1110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:33.115429 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:33.115430 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:37:33.115629 (+ 199us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:33.115660 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:37:33.115671 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:33.115768 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:33.116023 (+ 255us) server_negotiation.cc:299] Negotiation successful
0928 06:37:33.116055 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":83,"threads_started":1}
I20260928 06:37:33.116853 5645 catalog_manager.cc:1347] Generated new cluster ID: 52b80c93760c47dea864f25b6e08aa8b
I20260928 06:37:33.116904 5645 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:37:33.117522 17013 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:37:33.125048 5635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:33.121705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.60:58425 (local address 127.0.0.1:42487)
0928 06:37:33.121776 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:33.121778 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:33.121790 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:37:33.121814 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:33.121815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:33.121826 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:33.121903 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:33.122057 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:33.122788 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:33.123764 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:33.124084 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:33.124157 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:33.124795 (+ 638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:33.124796 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:33.124796 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:37:33.124800 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:33.124815 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:37:33.124819 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:33.124883 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:33.124965 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 06:37:33.124975 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 06:37:33.138432 5645 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:37:33.138936 5645 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:37:33.143472 5645 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b0f1dad41ea4ac2ac25af6f105ccdb8: Generated new TSK 0
I20260928 06:37:33.143649 5645 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:37:33.149652 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 5563
W20260928 06:37:33.154279 5548 connection.cc:535] client connection to 127.0.0.1:42487 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42487): Connection reset by peer (error 104)
2026-09-28T06:37:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1327 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577196946045-17013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:37:33 dist-test-slave-rrj5 krb5kdc[5662](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:37:33 dist-test-slave-rrj5 krb5kdc[5662](info): set up 2 sockets
Sep 28 06:37:33 dist-test-slave-rrj5 krb5kdc[5662](info): commencing operation
krb5kdc: starting...
W20260928 06:37:34.162539 17013 mini_kdc.cc:121] Time spent starting KDC: real 0.995s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Sep 28 06:37:34 dist-test-slave-rrj5 krb5kdc[5662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:37:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:37:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577196946045-17013-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577196946045-17013-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:37:34.278467 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577196946045-17013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577196946045-17013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577196946045-17013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37659
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577196946045-17013-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37659
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:37:34.358053 5675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:34.358242 5675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:34.358263 5675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:34.359305 5675 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:37:34.359339 5675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:34.359354 5675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:34.359365 5675 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:37:34.359377 5675 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:37:34.360697 5675 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577196946045-17013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577196946045-17013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37659
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577196946045-17013-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577196946045-17013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:37:34.360864 5675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:34.361014 5675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:34.365660 5683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:34.365705 5681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:34.365679 5680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:34.365785 5675 server_base.cc:1034] running on GCE node
I20260928 06:37:34.366109 5675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:34.366387 5675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:34.367527 5675 hybrid_clock.cc:648] HybridClock initialized: now 1790577454367517 us; error 27 us; skew 500 ppm
Sep 28 06:37:34 dist-test-slave-rrj5 krb5kdc[5662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:34.369879 5675 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:37:34.370327 5675 webserver.cc:460] Webserver started at http://127.0.0.1:35525/ using document root <none> and password file <none>
I20260928 06:37:34.370517 5675 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:34.370556 5675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:34.370656 5675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:37:34.371666 5675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577196946045-17013-0/minicluster-data/master-0/data/instance:
uuid: "122b323b41f945e7b9f35a364e81872d"
format_stamp: "Formatted at 2026-09-28 06:37:34 on dist-test-slave-rrj5"
server_key: "1334dcbd33fd04e42247960b8c79834f"
server_key_iv: "807129f344313b042128d2b9e47ec973"
server_key_version: "encryptionkey@0"
I20260928 06:37:34.371944 5675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577196946045-17013-0/minicluster-data/master-0/wal/instance:
uuid: "122b323b41f945e7b9f35a364e81872d"
format_stamp: "Formatted at 2026-09-28 06:37:34 on dist-test-slave-rrj5"
server_key: "1334dcbd33fd04e42247960b8c79834f"
server_key_iv: "807129f344313b042128d2b9e47ec973"
server_key_version: "encryptionkey@0"
I20260928 06:37:34.373131 5675 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 06:37:34.373813 5690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:34.373980 5675 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:34.374069 5675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577196946045-17013-0/minicluster-data/master-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577196946045-17013-0/minicluster-data/master-0/wal
uuid: "122b323b41f945e7b9f35a364e81872d"
format_stamp: "Formatted at 2026-09-28 06:37:34 on dist-test-slave-rrj5"
server_key: "1334dcbd33fd04e42247960b8c79834f"
server_key_iv: "807129f344313b042128d2b9e47ec973"
server_key_version: "encryptionkey@0"
I20260928 06:37:34.374140 5675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577196946045-17013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577196946045-17013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577196946045-17013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:34.377256 5675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:34.379009 5675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:34.379156 5675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:34.383219 5675 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37659
I20260928 06:37:34.383250 5742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37659 every 8 connection(s)
I20260928 06:37:34.383564 5675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
I20260928 06:37:34.384361 5743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:37:34.386637 5743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 122b323b41f945e7b9f35a364e81872d: Bootstrap starting.
I20260928 06:37:34.387200 5743 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 122b323b41f945e7b9f35a364e81872d: Neither blocks nor log segments found. Creating new log.
I20260928 06:37:34.387495 5743 log.cc:824] T 00000000000000000000000000000000 P 122b323b41f945e7b9f35a364e81872d: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:34.388180 5743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 122b323b41f945e7b9f35a364e81872d: No bootstrap required, opened a new log
I20260928 06:37:34.389809 5743 raft_consensus.cc:348] T 00000000000000000000000000000000 P 122b323b41f945e7b9f35a364e81872d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "122b323b41f945e7b9f35a364e81872d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37659 } }
I20260928 06:37:34.389936 5743 raft_consensus.cc:374] T 00000000000000000000000000000000 P 122b323b41f945e7b9f35a364e81872d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:37:34.389958 5743 raft_consensus.cc:729] T 00000000000000000000000000000000 P 122b323b41f945e7b9f35a364e81872d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 122b323b41f945e7b9f35a364e81872d, State: Initialized, Role: FOLLOWER
I20260928 06:37:34.390144 5743 consensus_queue.cc:260] T 00000000000000000000000000000000 P 122b323b41f945e7b9f35a364e81872d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "122b323b41f945e7b9f35a364e81872d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37659 } }
I20260928 06:37:34.390208 5743 raft_consensus.cc:388] T 00000000000000000000000000000000 P 122b323b41f945e7b9f35a364e81872d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:37:34.390228 5743 raft_consensus.cc:482] T 00000000000000000000000000000000 P 122b323b41f945e7b9f35a364e81872d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:37:34.390293 5743 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 122b323b41f945e7b9f35a364e81872d [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:37:34.390808 5743 raft_consensus.cc:504] T 00000000000000000000000000000000 P 122b323b41f945e7b9f35a364e81872d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "122b323b41f945e7b9f35a364e81872d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37659 } }
I20260928 06:37:34.390923 5743 leader_election.cc:302] T 00000000000000000000000000000000 P 122b323b41f945e7b9f35a364e81872d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 122b323b41f945e7b9f35a364e81872d; no voters:
I20260928 06:37:34.391105 5743 leader_election.cc:288] T 00000000000000000000000000000000 P 122b323b41f945e7b9f35a364e81872d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:37:34.391141 5746 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 122b323b41f945e7b9f35a364e81872d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:37:34.391302 5746 raft_consensus.cc:686] T 00000000000000000000000000000000 P 122b323b41f945e7b9f35a364e81872d [term 1 LEADER]: Becoming Leader. State: Replica: 122b323b41f945e7b9f35a364e81872d, State: Running, Role: LEADER
I20260928 06:37:34.391352 5743 sys_catalog.cc:564] T 00000000000000000000000000000000 P 122b323b41f945e7b9f35a364e81872d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:37:34.391450 5746 consensus_queue.cc:237] T 00000000000000000000000000000000 P 122b323b41f945e7b9f35a364e81872d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "122b323b41f945e7b9f35a364e81872d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37659 } }
I20260928 06:37:34.392124 5747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 122b323b41f945e7b9f35a364e81872d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "122b323b41f945e7b9f35a364e81872d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "122b323b41f945e7b9f35a364e81872d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37659 } } }
I20260928 06:37:34.392203 5747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 122b323b41f945e7b9f35a364e81872d [sys.catalog]: This master's current role is: LEADER
I20260928 06:37:34.392154 5748 sys_catalog.cc:455] T 00000000000000000000000000000000 P 122b323b41f945e7b9f35a364e81872d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 122b323b41f945e7b9f35a364e81872d. Latest consensus state: current_term: 1 leader_uuid: "122b323b41f945e7b9f35a364e81872d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "122b323b41f945e7b9f35a364e81872d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37659 } } }
I20260928 06:37:34.392242 5748 sys_catalog.cc:458] T 00000000000000000000000000000000 P 122b323b41f945e7b9f35a364e81872d [sys.catalog]: This master's current role is: LEADER
I20260928 06:37:34.392385 5756 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:37:34.392839 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 5675
I20260928 06:37:34.392939 5756 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:37:34.392971 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577196946045-17013-0/minicluster-data/master-0/wal/instance
I20260928 06:37:34.393175 17013 external_mini_cluster.cc:1428] Setting key 391ef69719d72ece086dbc21a653a965
I20260928 06:37:34.394613 5756 catalog_manager.cc:1347] Generated new cluster ID: 5cb197a2f91a486db554614b8d167cd4
I20260928 06:37:34.394671 5756 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:37:34.403810 5765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:34.393622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.60:38179 (local address 127.0.0.1:37659)
0928 06:37:34.402270 (+ 8648us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:34.402280 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:34.402317 (+ 37us) server_negotiation.cc:407] Connection header received
0928 06:37:34.402441 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:34.402447 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:34.402478 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:34.402627 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:34.403349 (+ 722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:34.403353 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:34.403354 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:37:34.403358 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:34.403398 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:37:34.403416 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:34.403485 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:34.403570 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 06:37:34.403614 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8589,"thread_start_us":69,"threads_started":1}
I20260928 06:37:34.404726 17013 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:37:34.407647 5756 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:37:34.407826 5756 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:37:34.408696 5765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:34.406260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.60:38935 (local address 127.0.0.1:37659)
0928 06:37:34.406295 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:34.406297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:34.407680 (+ 1383us) server_negotiation.cc:407] Connection header received
0928 06:37:34.407793 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:34.407794 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:34.407817 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:34.407887 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:34.408475 (+ 588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:34.408477 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:34.408477 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:37:34.408481 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:34.408495 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:37:34.408499 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:34.408564 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:34.408624 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 06:37:34.408637 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 06:37:34.411695 5756 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 122b323b41f945e7b9f35a364e81872d: Generated new TSK 0
I20260928 06:37:34.411849 5756 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:37:34.415517 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 5675
2026-09-28T06:37:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1265 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577196946045-17013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:37:34 dist-test-slave-rrj5 krb5kdc[5774](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:37:34 dist-test-slave-rrj5 krb5kdc[5774](info): set up 2 sockets
Sep 28 06:37:34 dist-test-slave-rrj5 krb5kdc[5774](info): commencing operation
krb5kdc: starting...
W20260928 06:37:35.446595 17013 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Sep 28 06:37:35 dist-test-slave-rrj5 krb5kdc[5774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:37:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:37:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577196946045-17013-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577196946045-17013-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:37:35.562444 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577196946045-17013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577196946045-17013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577196946045-17013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42655
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577196946045-17013-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42655
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:37:35.640846 5787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:35.641050 5787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:35.641072 5787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:35.642243 5787 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:37:35.642287 5787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:35.642300 5787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:35.642313 5787 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:37:35.642361 5787 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:37:35.643803 5787 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577196946045-17013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577196946045-17013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42655
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577196946045-17013-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577196946045-17013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:37:35.644001 5787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:35.644227 5787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:35.648859 5793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:35.648869 5792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:35.648876 5795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:35.648916 5787 server_base.cc:1034] running on GCE node
I20260928 06:37:35.649305 5787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:35.649689 5787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:35.650836 5787 hybrid_clock.cc:648] HybridClock initialized: now 1790577455650817 us; error 38 us; skew 500 ppm
Sep 28 06:37:35 dist-test-slave-rrj5 krb5kdc[5774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:35.656819 5787 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:37:35.657286 5787 webserver.cc:460] Webserver started at http://127.0.0.1:35361/ using document root <none> and password file <none>
I20260928 06:37:35.657518 5787 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:35.657604 5787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:35.657754 5787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:37:35.658756 5787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577196946045-17013-0/minicluster-data/master-0/data/instance:
uuid: "d46b3cecde204d70b066eede31af2157"
format_stamp: "Formatted at 2026-09-28 06:37:35 on dist-test-slave-rrj5"
server_key: "529bed61bdc63b973323e24540c8f3ea"
server_key_iv: "67215484cfd432562120b24c23833ab2"
server_key_version: "encryptionkey@0"
I20260928 06:37:35.659114 5787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577196946045-17013-0/minicluster-data/master-0/wal/instance:
uuid: "d46b3cecde204d70b066eede31af2157"
format_stamp: "Formatted at 2026-09-28 06:37:35 on dist-test-slave-rrj5"
server_key: "529bed61bdc63b973323e24540c8f3ea"
server_key_iv: "67215484cfd432562120b24c23833ab2"
server_key_version: "encryptionkey@0"
I20260928 06:37:35.660513 5787 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:37:35.661216 5802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:35.661391 5787 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:37:35.661489 5787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577196946045-17013-0/minicluster-data/master-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577196946045-17013-0/minicluster-data/master-0/wal
uuid: "d46b3cecde204d70b066eede31af2157"
format_stamp: "Formatted at 2026-09-28 06:37:35 on dist-test-slave-rrj5"
server_key: "529bed61bdc63b973323e24540c8f3ea"
server_key_iv: "67215484cfd432562120b24c23833ab2"
server_key_version: "encryptionkey@0"
I20260928 06:37:35.661576 5787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577196946045-17013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577196946045-17013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577196946045-17013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:35.684875 5787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:35.686786 5787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:35.686952 5787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:35.691138 5787 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42655
I20260928 06:37:35.691179 5854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42655 every 8 connection(s)
I20260928 06:37:35.691541 5787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
I20260928 06:37:35.692380 5855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:37:35.694639 5855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d46b3cecde204d70b066eede31af2157: Bootstrap starting.
I20260928 06:37:35.695148 5855 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d46b3cecde204d70b066eede31af2157: Neither blocks nor log segments found. Creating new log.
I20260928 06:37:35.695382 5855 log.cc:824] T 00000000000000000000000000000000 P d46b3cecde204d70b066eede31af2157: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:35.696007 5855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d46b3cecde204d70b066eede31af2157: No bootstrap required, opened a new log
I20260928 06:37:35.697036 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 5787
I20260928 06:37:35.697180 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577196946045-17013-0/minicluster-data/master-0/wal/instance
I20260928 06:37:35.697382 17013 external_mini_cluster.cc:1428] Setting key 78b1c74b97ec11bd1909c86f6ae2d9c0
I20260928 06:37:35.698635 5855 raft_consensus.cc:348] T 00000000000000000000000000000000 P d46b3cecde204d70b066eede31af2157 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d46b3cecde204d70b066eede31af2157" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42655 } }
I20260928 06:37:35.698809 5855 raft_consensus.cc:374] T 00000000000000000000000000000000 P d46b3cecde204d70b066eede31af2157 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:37:35.698851 5855 raft_consensus.cc:729] T 00000000000000000000000000000000 P d46b3cecde204d70b066eede31af2157 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d46b3cecde204d70b066eede31af2157, State: Initialized, Role: FOLLOWER
I20260928 06:37:35.698993 5855 consensus_queue.cc:260] T 00000000000000000000000000000000 P d46b3cecde204d70b066eede31af2157 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d46b3cecde204d70b066eede31af2157" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42655 } }
I20260928 06:37:35.699082 5855 raft_consensus.cc:388] T 00000000000000000000000000000000 P d46b3cecde204d70b066eede31af2157 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:37:35.699118 5855 raft_consensus.cc:482] T 00000000000000000000000000000000 P d46b3cecde204d70b066eede31af2157 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:37:35.699170 5855 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d46b3cecde204d70b066eede31af2157 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:37:35.699872 5855 raft_consensus.cc:504] T 00000000000000000000000000000000 P d46b3cecde204d70b066eede31af2157 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d46b3cecde204d70b066eede31af2157" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42655 } }
I20260928 06:37:35.699995 5855 leader_election.cc:302] T 00000000000000000000000000000000 P d46b3cecde204d70b066eede31af2157 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d46b3cecde204d70b066eede31af2157; no voters:
I20260928 06:37:35.700165 5855 leader_election.cc:288] T 00000000000000000000000000000000 P d46b3cecde204d70b066eede31af2157 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:37:35.700199 5860 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d46b3cecde204d70b066eede31af2157 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:37:35.700373 5860 raft_consensus.cc:686] T 00000000000000000000000000000000 P d46b3cecde204d70b066eede31af2157 [term 1 LEADER]: Becoming Leader. State: Replica: d46b3cecde204d70b066eede31af2157, State: Running, Role: LEADER
I20260928 06:37:35.700438 5855 sys_catalog.cc:564] T 00000000000000000000000000000000 P d46b3cecde204d70b066eede31af2157 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:37:35.700523 5860 consensus_queue.cc:237] T 00000000000000000000000000000000 P d46b3cecde204d70b066eede31af2157 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d46b3cecde204d70b066eede31af2157" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42655 } }
I20260928 06:37:35.700830 5860 sys_catalog.cc:455] T 00000000000000000000000000000000 P d46b3cecde204d70b066eede31af2157 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d46b3cecde204d70b066eede31af2157. Latest consensus state: current_term: 1 leader_uuid: "d46b3cecde204d70b066eede31af2157" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d46b3cecde204d70b066eede31af2157" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42655 } } }
I20260928 06:37:35.700840 5861 sys_catalog.cc:455] T 00000000000000000000000000000000 P d46b3cecde204d70b066eede31af2157 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d46b3cecde204d70b066eede31af2157" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d46b3cecde204d70b066eede31af2157" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42655 } } }
I20260928 06:37:35.700982 5861 sys_catalog.cc:458] T 00000000000000000000000000000000 P d46b3cecde204d70b066eede31af2157 [sys.catalog]: This master's current role is: LEADER
I20260928 06:37:35.701184 5860 sys_catalog.cc:458] T 00000000000000000000000000000000 P d46b3cecde204d70b066eede31af2157 [sys.catalog]: This master's current role is: LEADER
I20260928 06:37:35.701349 5867 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:37:35.701885 5867 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:37:35.703833 5867 catalog_manager.cc:1347] Generated new cluster ID: ea5dda238fb142f9b629c1a194d959b5
I20260928 06:37:35.703891 5867 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:37:35 dist-test-slave-rrj5 krb5kdc[5774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 06:37:35.708693 5859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:35.697977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.60:49701 (local address 127.0.0.1:42655)
0928 06:37:35.698174 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:35.698182 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:35.698209 (+ 27us) server_negotiation.cc:407] Connection header received
0928 06:37:35.698617 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:35.698623 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:35.698670 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:35.698779 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:35.699634 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:35.700899 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:35.702140 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:35.702745 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:35.702872 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:35.705564 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:35.705586 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:35.705586 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:35.705597 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:35.707204 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:35.707588 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:35.707591 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:35.707593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:35.707691 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:35.707919 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:35.707920 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:35.707920 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:35.708150 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:35.708331 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:35.708411 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 06:37:35.708522 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":100,"threads_started":1}
I20260928 06:37:35.709594 17013 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:37:35.716748 5859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:35.710882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.60:51807 (local address 127.0.0.1:42655)
0928 06:37:35.710922 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:35.710924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:35.711052 (+ 128us) server_negotiation.cc:407] Connection header received
0928 06:37:35.711122 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:35.711123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:35.711132 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:35.711187 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:35.711690 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:35.712152 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:35.712824 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:35.713048 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:35.713122 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:35.714271 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:35.714272 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:35.714273 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:35.714279 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:35.715631 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:35.715989 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:35.715990 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:35.715990 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:35.716047 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:35.716262 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:35.716263 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:35.716264 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:35.716369 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:35.716472 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:35.716589 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 06:37:35.716649 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:37:35.724709 5867 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:37:35.725229 5867 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:37:35.732877 5867 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d46b3cecde204d70b066eede31af2157: Generated new TSK 0
I20260928 06:37:35.733057 5867 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:37:35.735282 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 5787
W20260928 06:37:35.739909 5772 connection.cc:535] client connection to 127.0.0.1:42655 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42655): Connection reset by peer (error 104)
2026-09-28T06:37:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1320 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577196946045-17013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:37:35 dist-test-slave-rrj5 krb5kdc[5886](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:37:35 dist-test-slave-rrj5 krb5kdc[5886](info): set up 2 sockets
Sep 28 06:37:35 dist-test-slave-rrj5 krb5kdc[5886](info): commencing operation
krb5kdc: starting...
W20260928 06:37:36.826527 17013 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Sep 28 06:37:36 dist-test-slave-rrj5 krb5kdc[5886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:37:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:37:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577196946045-17013-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577196946045-17013-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:37:36.945096 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577196946045-17013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577196946045-17013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577196946045-17013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46609
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577196946045-17013-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46609
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:37:37.023962 5899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:37.024142 5899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:37.024163 5899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:37.025208 5899 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:37:37.025243 5899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:37.025257 5899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:37.025270 5899 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:37:37.025282 5899 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:37:37.026654 5899 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577196946045-17013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577196946045-17013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46609
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577196946045-17013-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5899
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577196946045-17013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:37:37.026818 5899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:37.026980 5899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:37.031586 5907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:37.031595 5905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:37.031600 5904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:37.031883 5899 server_base.cc:1034] running on GCE node
I20260928 06:37:37.032083 5899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:37.032303 5899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:37.033439 5899 hybrid_clock.cc:648] HybridClock initialized: now 1790577457033425 us; error 28 us; skew 500 ppm
Sep 28 06:37:37 dist-test-slave-rrj5 krb5kdc[5886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577457, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:37.035888 5899 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:37:37.036257 5899 webserver.cc:460] Webserver started at http://127.0.0.1:38355/ using document root <none> and password file <none>
I20260928 06:37:37.036442 5899 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:37.036480 5899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:37.036568 5899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:37:37.037498 5899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577196946045-17013-0/minicluster-data/master-0/data/instance:
uuid: "6010de9a728f4dabb3bbb03f551178fa"
format_stamp: "Formatted at 2026-09-28 06:37:37 on dist-test-slave-rrj5"
server_key: "88d9946be3a4f1283713a6370d231b6e"
server_key_iv: "73a61abf025c513d0123bc5a101e91cd"
server_key_version: "encryptionkey@0"
I20260928 06:37:37.037902 5899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577196946045-17013-0/minicluster-data/master-0/wal/instance:
uuid: "6010de9a728f4dabb3bbb03f551178fa"
format_stamp: "Formatted at 2026-09-28 06:37:37 on dist-test-slave-rrj5"
server_key: "88d9946be3a4f1283713a6370d231b6e"
server_key_iv: "73a61abf025c513d0123bc5a101e91cd"
server_key_version: "encryptionkey@0"
I20260928 06:37:37.039151 5899 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:37:37.039829 5914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:37.039988 5899 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:37:37.040092 5899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577196946045-17013-0/minicluster-data/master-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577196946045-17013-0/minicluster-data/master-0/wal
uuid: "6010de9a728f4dabb3bbb03f551178fa"
format_stamp: "Formatted at 2026-09-28 06:37:37 on dist-test-slave-rrj5"
server_key: "88d9946be3a4f1283713a6370d231b6e"
server_key_iv: "73a61abf025c513d0123bc5a101e91cd"
server_key_version: "encryptionkey@0"
I20260928 06:37:37.040162 5899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577196946045-17013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577196946045-17013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577196946045-17013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:37.050942 5899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:37.053122 5899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:37.053273 5899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:37.057858 5899 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46609
I20260928 06:37:37.057929 5966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46609 every 8 connection(s)
I20260928 06:37:37.058391 5899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
I20260928 06:37:37.059275 5967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:37:37.059695 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 5899
I20260928 06:37:37.059923 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577196946045-17013-0/minicluster-data/master-0/wal/instance
I20260928 06:37:37.060178 17013 external_mini_cluster.cc:1428] Setting key a2f3be41c98edb021d398c1d27093144
I20260928 06:37:37.063220 5967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6010de9a728f4dabb3bbb03f551178fa: Bootstrap starting.
I20260928 06:37:37.063877 5967 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6010de9a728f4dabb3bbb03f551178fa: Neither blocks nor log segments found. Creating new log.
I20260928 06:37:37.064167 5967 log.cc:824] T 00000000000000000000000000000000 P 6010de9a728f4dabb3bbb03f551178fa: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:37.064978 5967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6010de9a728f4dabb3bbb03f551178fa: No bootstrap required, opened a new log
I20260928 06:37:37.067203 5970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:37.060735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.60:51087 (local address 127.0.0.1:46609)
0928 06:37:37.060930 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:37.060940 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:37.061846 (+ 906us) server_negotiation.cc:407] Connection header received
0928 06:37:37.062373 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:37.062382 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:37.062456 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:37.062605 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:37.062858 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:37.064189 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:37.065306 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:37.065833 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:37.065937 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:37.066631 (+ 694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:37.066635 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:37.066635 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:37:37.066793 (+ 158us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:37.066821 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:37:37.066832 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:37.066897 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:37.067028 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 06:37:37.067055 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":86,"threads_started":1}
I20260928 06:37:37.067448 5967 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6010de9a728f4dabb3bbb03f551178fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6010de9a728f4dabb3bbb03f551178fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46609 } }
I20260928 06:37:37.067629 5967 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6010de9a728f4dabb3bbb03f551178fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:37:37.067669 5967 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6010de9a728f4dabb3bbb03f551178fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6010de9a728f4dabb3bbb03f551178fa, State: Initialized, Role: FOLLOWER
I20260928 06:37:37.067796 5967 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6010de9a728f4dabb3bbb03f551178fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6010de9a728f4dabb3bbb03f551178fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46609 } }
I20260928 06:37:37.067875 5967 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6010de9a728f4dabb3bbb03f551178fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:37:37.067916 5967 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6010de9a728f4dabb3bbb03f551178fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:37:37.067970 5967 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6010de9a728f4dabb3bbb03f551178fa [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:37:37.068149 17013 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:37:37.068543 5967 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6010de9a728f4dabb3bbb03f551178fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6010de9a728f4dabb3bbb03f551178fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46609 } }
I20260928 06:37:37.068652 5967 leader_election.cc:302] T 00000000000000000000000000000000 P 6010de9a728f4dabb3bbb03f551178fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6010de9a728f4dabb3bbb03f551178fa; no voters:
I20260928 06:37:37.068823 5967 leader_election.cc:288] T 00000000000000000000000000000000 P 6010de9a728f4dabb3bbb03f551178fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:37:37.068856 5972 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6010de9a728f4dabb3bbb03f551178fa [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:37:37.069007 5972 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6010de9a728f4dabb3bbb03f551178fa [term 1 LEADER]: Becoming Leader. State: Replica: 6010de9a728f4dabb3bbb03f551178fa, State: Running, Role: LEADER
I20260928 06:37:37.069109 5967 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6010de9a728f4dabb3bbb03f551178fa [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:37:37.069146 5972 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6010de9a728f4dabb3bbb03f551178fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6010de9a728f4dabb3bbb03f551178fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46609 } }
I20260928 06:37:37.069598 5973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6010de9a728f4dabb3bbb03f551178fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6010de9a728f4dabb3bbb03f551178fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6010de9a728f4dabb3bbb03f551178fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46609 } } }
I20260928 06:37:37.069746 5973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6010de9a728f4dabb3bbb03f551178fa [sys.catalog]: This master's current role is: LEADER
I20260928 06:37:37.069969 5973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6010de9a728f4dabb3bbb03f551178fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6010de9a728f4dabb3bbb03f551178fa. Latest consensus state: current_term: 1 leader_uuid: "6010de9a728f4dabb3bbb03f551178fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6010de9a728f4dabb3bbb03f551178fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46609 } } }
I20260928 06:37:37.070032 5973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6010de9a728f4dabb3bbb03f551178fa [sys.catalog]: This master's current role is: LEADER
I20260928 06:37:37.070036 5986 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:37:37.070601 5986 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:37:37.072160 5986 catalog_manager.cc:1347] Generated new cluster ID: 8d669451a950401f92c005f29be7e1ad
I20260928 06:37:37.072216 5986 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:37:37.073781 5970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:37.069599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.60:36027 (local address 127.0.0.1:46609)
0928 06:37:37.069632 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:37.069634 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:37.070660 (+ 1026us) server_negotiation.cc:407] Connection header received
0928 06:37:37.070742 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:37.070743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:37.070754 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:37.070786 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:37.071033 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:37.071698 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:37.072602 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:37.072824 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:37.072895 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:37.073539 (+ 644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:37.073540 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:37.073541 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:37:37.073545 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:37.073559 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:37:37.073563 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:37.073637 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:37.073698 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 06:37:37.073706 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 06:37:37.094401 5986 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:37:37.094942 5986 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:37:37.097986 5986 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6010de9a728f4dabb3bbb03f551178fa: Generated new TSK 0
I20260928 06:37:37.098172 5986 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:37:37.105549 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 5899
W20260928 06:37:37.110203 5884 connection.cc:535] client connection to 127.0.0.1:46609 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46609): Connection reset by peer (error 104)
2026-09-28T06:37:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1370 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577196946045-17013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:37:37 dist-test-slave-rrj5 krb5kdc[5998](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:37:37 dist-test-slave-rrj5 krb5kdc[5998](info): set up 2 sockets
Sep 28 06:37:37 dist-test-slave-rrj5 krb5kdc[5998](info): commencing operation
krb5kdc: starting...
W20260928 06:37:38.146518 17013 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Sep 28 06:37:38 dist-test-slave-rrj5 krb5kdc[5998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:37:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:37:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577196946045-17013-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577196946045-17013-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:37:38.264811 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577196946045-17013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577196946045-17013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577196946045-17013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40641
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577196946045-17013-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40641
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:37:38.345412 6011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:38.345669 6011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:38.345710 6011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:38.346841 6011 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:37:38.346913 6011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:38.346938 6011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:38.346966 6011 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:37:38.346993 6011 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:37:38.348474 6011 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577196946045-17013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577196946045-17013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40641
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577196946045-17013-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577196946045-17013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:37:38.348737 6011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:38.348959 6011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:38.353746 6019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:38.353753 6016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:38.353788 6017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:38.353776 6011 server_base.cc:1034] running on GCE node
I20260928 06:37:38.354173 6011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:38.354460 6011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:38.355599 6011 hybrid_clock.cc:648] HybridClock initialized: now 1790577458355583 us; error 33 us; skew 500 ppm
Sep 28 06:37:38 dist-test-slave-rrj5 krb5kdc[5998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577458, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:38.358067 6011 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:37:38.358539 6011 webserver.cc:460] Webserver started at http://127.0.0.1:42533/ using document root <none> and password file <none>
I20260928 06:37:38.358790 6011 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:38.358836 6011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:38.359004 6011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:37:38.360054 6011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577196946045-17013-0/minicluster-data/master-0/data/instance:
uuid: "d4e0698a903140bfafa4317c8cfc430a"
format_stamp: "Formatted at 2026-09-28 06:37:38 on dist-test-slave-rrj5"
server_key: "32f923ceefe3840566232701a9af8654"
server_key_iv: "380db8c1b38410d2b03030a56bc14cd7"
server_key_version: "encryptionkey@0"
I20260928 06:37:38.360405 6011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577196946045-17013-0/minicluster-data/master-0/wal/instance:
uuid: "d4e0698a903140bfafa4317c8cfc430a"
format_stamp: "Formatted at 2026-09-28 06:37:38 on dist-test-slave-rrj5"
server_key: "32f923ceefe3840566232701a9af8654"
server_key_iv: "380db8c1b38410d2b03030a56bc14cd7"
server_key_version: "encryptionkey@0"
I20260928 06:37:38.361738 6011 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 06:37:38.362442 6026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:38.362622 6011 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:37:38.362710 6011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577196946045-17013-0/minicluster-data/master-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577196946045-17013-0/minicluster-data/master-0/wal
uuid: "d4e0698a903140bfafa4317c8cfc430a"
format_stamp: "Formatted at 2026-09-28 06:37:38 on dist-test-slave-rrj5"
server_key: "32f923ceefe3840566232701a9af8654"
server_key_iv: "380db8c1b38410d2b03030a56bc14cd7"
server_key_version: "encryptionkey@0"
I20260928 06:37:38.362782 6011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577196946045-17013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577196946045-17013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577196946045-17013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:38.366084 6011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:38.367808 6011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:38.367955 6011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:38.371941 6011 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40641
I20260928 06:37:38.371973 6078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40641 every 8 connection(s)
I20260928 06:37:38.372289 6011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
I20260928 06:37:38.373116 6079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:37:38.375317 6079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4e0698a903140bfafa4317c8cfc430a: Bootstrap starting.
I20260928 06:37:38.375865 6079 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4e0698a903140bfafa4317c8cfc430a: Neither blocks nor log segments found. Creating new log.
I20260928 06:37:38.376145 6079 log.cc:824] T 00000000000000000000000000000000 P d4e0698a903140bfafa4317c8cfc430a: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:38.376773 6079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4e0698a903140bfafa4317c8cfc430a: No bootstrap required, opened a new log
I20260928 06:37:38.378404 6079 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4e0698a903140bfafa4317c8cfc430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e0698a903140bfafa4317c8cfc430a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40641 } }
I20260928 06:37:38.378536 6079 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4e0698a903140bfafa4317c8cfc430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:37:38.378579 6079 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4e0698a903140bfafa4317c8cfc430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4e0698a903140bfafa4317c8cfc430a, State: Initialized, Role: FOLLOWER
I20260928 06:37:38.378750 6079 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4e0698a903140bfafa4317c8cfc430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e0698a903140bfafa4317c8cfc430a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40641 } }
I20260928 06:37:38.378829 6079 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4e0698a903140bfafa4317c8cfc430a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:37:38.378875 6079 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4e0698a903140bfafa4317c8cfc430a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:37:38.378926 6079 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4e0698a903140bfafa4317c8cfc430a [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:37:38.379419 6079 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4e0698a903140bfafa4317c8cfc430a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e0698a903140bfafa4317c8cfc430a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40641 } }
I20260928 06:37:38.379525 6079 leader_election.cc:302] T 00000000000000000000000000000000 P d4e0698a903140bfafa4317c8cfc430a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4e0698a903140bfafa4317c8cfc430a; no voters:
I20260928 06:37:38.379693 6079 leader_election.cc:288] T 00000000000000000000000000000000 P d4e0698a903140bfafa4317c8cfc430a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:37:38.379724 6082 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4e0698a903140bfafa4317c8cfc430a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:37:38.379877 6082 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4e0698a903140bfafa4317c8cfc430a [term 1 LEADER]: Becoming Leader. State: Replica: d4e0698a903140bfafa4317c8cfc430a, State: Running, Role: LEADER
I20260928 06:37:38.379945 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 6011
I20260928 06:37:38.379944 6079 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4e0698a903140bfafa4317c8cfc430a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:37:38.380064 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577196946045-17013-0/minicluster-data/master-0/wal/instance
I20260928 06:37:38.380074 6082 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4e0698a903140bfafa4317c8cfc430a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e0698a903140bfafa4317c8cfc430a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40641 } }
I20260928 06:37:38.380244 17013 external_mini_cluster.cc:1428] Setting key 18d309e4c5c9ae2f4c090d2b8385ac7e
I20260928 06:37:38.380795 6084 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4e0698a903140bfafa4317c8cfc430a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4e0698a903140bfafa4317c8cfc430a. Latest consensus state: current_term: 1 leader_uuid: "d4e0698a903140bfafa4317c8cfc430a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e0698a903140bfafa4317c8cfc430a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40641 } } }
I20260928 06:37:38.380946 6084 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4e0698a903140bfafa4317c8cfc430a [sys.catalog]: This master's current role is: LEADER
I20260928 06:37:38.380962 6083 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4e0698a903140bfafa4317c8cfc430a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4e0698a903140bfafa4317c8cfc430a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e0698a903140bfafa4317c8cfc430a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40641 } } }
I20260928 06:37:38.381096 6083 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4e0698a903140bfafa4317c8cfc430a [sys.catalog]: This master's current role is: LEADER
W20260928 06:37:38.382249 6099 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d4e0698a903140bfafa4317c8cfc430a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:37:38.382354 6099 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:37:38.382457 6095 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:37:38.382691 6095 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:37:38.382870 6092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:38.380920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.60:36765 (local address 127.0.0.1:40641)
0928 06:37:38.381109 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:38.381119 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:38.381245 (+ 126us) server_negotiation.cc:407] Connection header received
0928 06:37:38.381484 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:38.381492 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:38.381524 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:38.381995 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:38.382409 (+ 414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:38.382412 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:38.382413 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:37:38.382416 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:38.382451 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:37:38.382466 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:38.382535 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:38.382608 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 06:37:38.382653 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":84,"threads_started":1}
I20260928 06:37:38.383831 17013 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:37:38.384310 6095 catalog_manager.cc:1347] Generated new cluster ID: 8c5a1815314843bda2694040e858f357
I20260928 06:37:38.384363 6095 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:37:38.386427 6092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:38.385273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.60:34617 (local address 127.0.0.1:40641)
0928 06:37:38.385324 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:38.385326 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:38.385432 (+ 106us) server_negotiation.cc:407] Connection header received
0928 06:37:38.385510 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:38.385511 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:38.385517 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:38.385561 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:38.386182 (+ 621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:38.386184 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:38.386184 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:37:38.386189 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:38.386203 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:37:38.386207 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:38.386265 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:38.386320 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 06:37:38.386334 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 06:37:38.400312 6095 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:37:38.400482 6095 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:37:38.409140 6095 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4e0698a903140bfafa4317c8cfc430a: Generated new TSK 0
I20260928 06:37:38.409323 6095 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:37:38.413216 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 6011
2026-09-28T06:37:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1309 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577196946045-17013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:37:38 dist-test-slave-rrj5 krb5kdc[6110](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:37:38 dist-test-slave-rrj5 krb5kdc[6110](info): set up 2 sockets
Sep 28 06:37:38 dist-test-slave-rrj5 krb5kdc[6110](info): commencing operation
krb5kdc: starting...
W20260928 06:37:39.446952 17013 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Sep 28 06:37:39 dist-test-slave-rrj5 krb5kdc[6110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577459, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:37:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:37:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.157.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab.
Entry for principal kudu/127.16.157.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab.
I20260928 06:37:39.563694 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577196946045-17013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577196946045-17013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577196946045-17013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:45827
--webserver_interface=127.16.157.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=kudu/127.16.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:36665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.157.126:45827
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:37:39.643591 6123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:39.643790 6123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:39.643813 6123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:39.644876 6123 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:37:39.644912 6123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:39.644925 6123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:39.644938 6123 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:37:39.644950 6123 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:37:39.646385 6123 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:36665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577196946045-17013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577196946045-17013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.157.126:45827
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=kudu/127.16.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:45827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6123
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577196946045-17013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:37:39.646570 6123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:39.646737 6123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:39.651629 6129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:39.651590 6128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:39.651824 6123 server_base.cc:1034] running on GCE node
W20260928 06:37:39.651590 6131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:39.652117 6123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:39.652418 6123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:39.653559 6123 hybrid_clock.cc:648] HybridClock initialized: now 1790577459653543 us; error 35 us; skew 500 ppm
Sep 28 06:37:39 dist-test-slave-rrj5 krb5kdc[6110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577459, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:39.656011 6123 init.cc:377] Logged in from keytab as kudu/127.16.157.126@KRBTEST.COM (short username kudu)
I20260928 06:37:39.656487 6123 webserver.cc:460] Webserver started at http://127.16.157.126:45243/ using document root <none> and password file <none>
I20260928 06:37:39.656733 6123 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:39.656796 6123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:39.656941 6123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:37:39.658074 6123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577196946045-17013-0/minicluster-data/master-0/data/instance:
uuid: "f3a1a114e25444558f75e2d2574b9415"
format_stamp: "Formatted at 2026-09-28 06:37:39 on dist-test-slave-rrj5"
server_key: "c663b34a047859265166688d683e0c31"
server_key_iv: "5d0169768930df73f0666a848d71e512"
server_key_version: "encryptionkey@0"
I20260928 06:37:39.658424 6123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577196946045-17013-0/minicluster-data/master-0/wal/instance:
uuid: "f3a1a114e25444558f75e2d2574b9415"
format_stamp: "Formatted at 2026-09-28 06:37:39 on dist-test-slave-rrj5"
server_key: "c663b34a047859265166688d683e0c31"
server_key_iv: "5d0169768930df73f0666a848d71e512"
server_key_version: "encryptionkey@0"
I20260928 06:37:39.659694 6123 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:37:39.660454 6138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:39.660615 6123 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:37:39.660715 6123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577196946045-17013-0/minicluster-data/master-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577196946045-17013-0/minicluster-data/master-0/wal
uuid: "f3a1a114e25444558f75e2d2574b9415"
format_stamp: "Formatted at 2026-09-28 06:37:39 on dist-test-slave-rrj5"
server_key: "c663b34a047859265166688d683e0c31"
server_key_iv: "5d0169768930df73f0666a848d71e512"
server_key_version: "encryptionkey@0"
I20260928 06:37:39.660785 6123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577196946045-17013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577196946045-17013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577196946045-17013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:39.680706 6123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:39.682667 6123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:39.682832 6123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:39.687047 6123 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.126:45827
I20260928 06:37:39.687083 6190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.126:45827 every 8 connection(s)
I20260928 06:37:39.687449 6123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
I20260928 06:37:39.688361 6191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:37:39.688364 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 6123
I20260928 06:37:39.688481 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577196946045-17013-0/minicluster-data/master-0/wal/instance
I20260928 06:37:39.688680 17013 external_mini_cluster.cc:1428] Setting key ec4999602e52730c7b4c42a74214261b
I20260928 06:37:39.691622 6191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3a1a114e25444558f75e2d2574b9415: Bootstrap starting.
I20260928 06:37:39.692337 6191 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3a1a114e25444558f75e2d2574b9415: Neither blocks nor log segments found. Creating new log.
I20260928 06:37:39.692638 6191 log.cc:824] T 00000000000000000000000000000000 P f3a1a114e25444558f75e2d2574b9415: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:39.693347 6191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3a1a114e25444558f75e2d2574b9415: No bootstrap required, opened a new log
Sep 28 06:37:39 dist-test-slave-rrj5 krb5kdc[6110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577459, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:37:39.695902 6191 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3a1a114e25444558f75e2d2574b9415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3a1a114e25444558f75e2d2574b9415" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 45827 } }
I20260928 06:37:39.696098 6191 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3a1a114e25444558f75e2d2574b9415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:37:39.696138 6191 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3a1a114e25444558f75e2d2574b9415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3a1a114e25444558f75e2d2574b9415, State: Initialized, Role: FOLLOWER
I20260928 06:37:39.696280 6191 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3a1a114e25444558f75e2d2574b9415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3a1a114e25444558f75e2d2574b9415" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 45827 } }
I20260928 06:37:39.696384 6191 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3a1a114e25444558f75e2d2574b9415 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:37:39.696439 6191 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3a1a114e25444558f75e2d2574b9415 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:37:39.696497 6191 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3a1a114e25444558f75e2d2574b9415 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:37:39.697036 6191 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3a1a114e25444558f75e2d2574b9415 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3a1a114e25444558f75e2d2574b9415" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 45827 } }
I20260928 06:37:39.697140 6191 leader_election.cc:302] T 00000000000000000000000000000000 P f3a1a114e25444558f75e2d2574b9415 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3a1a114e25444558f75e2d2574b9415; no voters:
I20260928 06:37:39.697309 6191 leader_election.cc:288] T 00000000000000000000000000000000 P f3a1a114e25444558f75e2d2574b9415 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:37:39.697341 6196 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3a1a114e25444558f75e2d2574b9415 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:37:39.697456 6196 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3a1a114e25444558f75e2d2574b9415 [term 1 LEADER]: Becoming Leader. State: Replica: f3a1a114e25444558f75e2d2574b9415, State: Running, Role: LEADER
I20260928 06:37:39.697589 6191 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3a1a114e25444558f75e2d2574b9415 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:37:39.697613 6196 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3a1a114e25444558f75e2d2574b9415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3a1a114e25444558f75e2d2574b9415" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 45827 } }
I20260928 06:37:39.698040 6197 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3a1a114e25444558f75e2d2574b9415 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3a1a114e25444558f75e2d2574b9415" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3a1a114e25444558f75e2d2574b9415" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 45827 } } }
I20260928 06:37:39.698132 6197 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3a1a114e25444558f75e2d2574b9415 [sys.catalog]: This master's current role is: LEADER
I20260928 06:37:39.698297 6198 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3a1a114e25444558f75e2d2574b9415 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3a1a114e25444558f75e2d2574b9415. Latest consensus state: current_term: 1 leader_uuid: "f3a1a114e25444558f75e2d2574b9415" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3a1a114e25444558f75e2d2574b9415" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 45827 } } }
I20260928 06:37:39.698379 6198 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3a1a114e25444558f75e2d2574b9415 [sys.catalog]: This master's current role is: LEADER
I20260928 06:37:39.698792 6203 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:37:39.699813 6203 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:37:39.700134 6194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:39.689204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49115 (local address 127.16.157.126:45827)
0928 06:37:39.689369 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:39.689378 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:39.689412 (+ 34us) server_negotiation.cc:407] Connection header received
0928 06:37:39.689933 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:39.689940 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:39.689990 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:39.690111 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:39.690927 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:39.691864 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:39.692958 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:39.693423 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:39.693550 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:39.696033 (+ 2483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:39.696037 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:39.696038 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:39.696049 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:39.698046 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:39.698575 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:39.698577 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:39.698578 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:39.698695 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:39.699049 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:39.699051 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:39.699051 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:39.699312 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:39.699778 (+ 466us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:39.699865 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 06:37:39.699990 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":76,"threads_started":1}
I20260928 06:37:39.701166 17013 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:37:39.701444 6203 catalog_manager.cc:1347] Generated new cluster ID: aa807fa101fe42e098b59d338e666503
I20260928 06:37:39.701495 6203 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:37:39.709918 6194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:39.702630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47627 (local address 127.16.157.126:45827)
0928 06:37:39.702682 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:39.702684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:39.702753 (+ 69us) server_negotiation.cc:407] Connection header received
0928 06:37:39.702829 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:39.702829 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:39.702838 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:39.702887 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:39.703521 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:39.704203 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:39.705309 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:39.705662 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:39.705716 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:39.707062 (+ 1346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:39.707063 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:39.707064 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:39.707070 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:39.708607 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:39.709022 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:39.709023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:39.709023 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:39.709075 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:39.709308 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:39.709309 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:39.709309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:39.709418 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:39.709499 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:39.709685 (+ 186us) server_negotiation.cc:299] Negotiation successful
0928 06:37:39.709743 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 06:37:39.715435 6203 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:37:39.716109 6203 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:37:39.736997 6203 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f3a1a114e25444558f75e2d2574b9415: Generated new TSK 0
I20260928 06:37:39.737208 6203 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:37:39.739365 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 6123
W20260928 06:37:39.743808 6108 connection.cc:535] client connection to 127.16.157.126:45827 recv error: Network error: failed to read from TLS socket (remote: 127.16.157.126:45827): Connection reset by peer (error 104)
2026-09-28T06:37:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1323 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577196946045-17013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:37:39 dist-test-slave-rrj5 krb5kdc[6222](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:37:39 dist-test-slave-rrj5 krb5kdc[6222](info): set up 2 sockets
Sep 28 06:37:39 dist-test-slave-rrj5 krb5kdc[6222](info): commencing operation
krb5kdc: starting...
W20260928 06:37:40.762452 17013 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Sep 28 06:37:40 dist-test-slave-rrj5 krb5kdc[6222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:37:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:37:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.157.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab.
Entry for principal kudu/127.16.157.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab.
I20260928 06:37:40.878242 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577196946045-17013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577196946045-17013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577196946045-17013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:45425
--webserver_interface=127.16.157.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=kudu/127.16.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:35141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.157.126:45425
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:37:40.959262 6235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:40.959461 6235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:40.959493 6235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:40.960623 6235 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:37:40.960665 6235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:40.960680 6235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:40.960692 6235 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:37:40.960704 6235 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:37:40.962162 6235 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:35141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577196946045-17013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577196946045-17013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.157.126:45425
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=kudu/127.16.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:45425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577196946045-17013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:37:40.962402 6235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:40.962625 6235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:40.967514 6243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:40.967538 6240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:40.967581 6241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:40.967628 6235 server_base.cc:1034] running on GCE node
I20260928 06:37:40.967960 6235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:40.968287 6235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:40.969442 6235 hybrid_clock.cc:648] HybridClock initialized: now 1790577460969486 us; error 87 us; skew 500 ppm
Sep 28 06:37:40 dist-test-slave-rrj5 krb5kdc[6222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577460, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:40.975981 6235 init.cc:377] Logged in from keytab as kudu/127.16.157.126@KRBTEST.COM (short username kudu)
I20260928 06:37:40.976434 6235 webserver.cc:460] Webserver started at http://127.16.157.126:39475/ using document root <none> and password file <none>
I20260928 06:37:40.976660 6235 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:40.976725 6235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:40.976867 6235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:37:40.977974 6235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577196946045-17013-0/minicluster-data/master-0/data/instance:
uuid: "e97d719a7c094de29399d43093cee92a"
format_stamp: "Formatted at 2026-09-28 06:37:40 on dist-test-slave-rrj5"
server_key: "c9d2374fc0544f41ed1ae74a0ef7aeb3"
server_key_iv: "2ef4edd33dc02b382d3e4db4a501621c"
server_key_version: "encryptionkey@0"
I20260928 06:37:40.978335 6235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577196946045-17013-0/minicluster-data/master-0/wal/instance:
uuid: "e97d719a7c094de29399d43093cee92a"
format_stamp: "Formatted at 2026-09-28 06:37:40 on dist-test-slave-rrj5"
server_key: "c9d2374fc0544f41ed1ae74a0ef7aeb3"
server_key_iv: "2ef4edd33dc02b382d3e4db4a501621c"
server_key_version: "encryptionkey@0"
I20260928 06:37:40.979619 6235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:37:40.980300 6250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:40.980461 6235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:37:40.980561 6235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577196946045-17013-0/minicluster-data/master-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577196946045-17013-0/minicluster-data/master-0/wal
uuid: "e97d719a7c094de29399d43093cee92a"
format_stamp: "Formatted at 2026-09-28 06:37:40 on dist-test-slave-rrj5"
server_key: "c9d2374fc0544f41ed1ae74a0ef7aeb3"
server_key_iv: "2ef4edd33dc02b382d3e4db4a501621c"
server_key_version: "encryptionkey@0"
I20260928 06:37:40.980631 6235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577196946045-17013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577196946045-17013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577196946045-17013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:40.998276 6235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:41.000425 6235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:41.000581 6235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:41.004499 6235 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.126:45425
I20260928 06:37:41.004532 6302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.126:45425 every 8 connection(s)
I20260928 06:37:41.004856 6235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
I20260928 06:37:41.005736 6303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:37:41.007877 6303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e97d719a7c094de29399d43093cee92a: Bootstrap starting.
I20260928 06:37:41.008445 6303 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e97d719a7c094de29399d43093cee92a: Neither blocks nor log segments found. Creating new log.
I20260928 06:37:41.008720 6303 log.cc:824] T 00000000000000000000000000000000 P e97d719a7c094de29399d43093cee92a: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:41.009354 6303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e97d719a7c094de29399d43093cee92a: No bootstrap required, opened a new log
I20260928 06:37:41.010928 6303 raft_consensus.cc:348] T 00000000000000000000000000000000 P e97d719a7c094de29399d43093cee92a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e97d719a7c094de29399d43093cee92a" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 45425 } }
I20260928 06:37:41.011052 6303 raft_consensus.cc:374] T 00000000000000000000000000000000 P e97d719a7c094de29399d43093cee92a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:37:41.011073 6303 raft_consensus.cc:729] T 00000000000000000000000000000000 P e97d719a7c094de29399d43093cee92a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e97d719a7c094de29399d43093cee92a, State: Initialized, Role: FOLLOWER
I20260928 06:37:41.011191 6303 consensus_queue.cc:260] T 00000000000000000000000000000000 P e97d719a7c094de29399d43093cee92a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e97d719a7c094de29399d43093cee92a" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 45425 } }
I20260928 06:37:41.011266 6303 raft_consensus.cc:388] T 00000000000000000000000000000000 P e97d719a7c094de29399d43093cee92a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:37:41.011288 6303 raft_consensus.cc:482] T 00000000000000000000000000000000 P e97d719a7c094de29399d43093cee92a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:37:41.011315 6303 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e97d719a7c094de29399d43093cee92a [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:37:41.012065 6303 raft_consensus.cc:504] T 00000000000000000000000000000000 P e97d719a7c094de29399d43093cee92a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e97d719a7c094de29399d43093cee92a" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 45425 } }
I20260928 06:37:41.012149 6303 leader_election.cc:302] T 00000000000000000000000000000000 P e97d719a7c094de29399d43093cee92a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e97d719a7c094de29399d43093cee92a; no voters:
I20260928 06:37:41.012264 6303 leader_election.cc:288] T 00000000000000000000000000000000 P e97d719a7c094de29399d43093cee92a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:37:41.012326 6306 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e97d719a7c094de29399d43093cee92a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:37:41.012511 6303 sys_catalog.cc:564] T 00000000000000000000000000000000 P e97d719a7c094de29399d43093cee92a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:37:41.012526 6306 raft_consensus.cc:686] T 00000000000000000000000000000000 P e97d719a7c094de29399d43093cee92a [term 1 LEADER]: Becoming Leader. State: Replica: e97d719a7c094de29399d43093cee92a, State: Running, Role: LEADER
I20260928 06:37:41.012734 6306 consensus_queue.cc:237] T 00000000000000000000000000000000 P e97d719a7c094de29399d43093cee92a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e97d719a7c094de29399d43093cee92a" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 45425 } }
I20260928 06:37:41.012866 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 6235
I20260928 06:37:41.012977 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577196946045-17013-0/minicluster-data/master-0/wal/instance
I20260928 06:37:41.013208 17013 external_mini_cluster.cc:1428] Setting key e3f81d65ea7e656bc730cd6024dd8499
I20260928 06:37:41.013185 6307 sys_catalog.cc:455] T 00000000000000000000000000000000 P e97d719a7c094de29399d43093cee92a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e97d719a7c094de29399d43093cee92a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e97d719a7c094de29399d43093cee92a" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 45425 } } }
I20260928 06:37:41.013257 6307 sys_catalog.cc:458] T 00000000000000000000000000000000 P e97d719a7c094de29399d43093cee92a [sys.catalog]: This master's current role is: LEADER
I20260928 06:37:41.013460 6308 sys_catalog.cc:455] T 00000000000000000000000000000000 P e97d719a7c094de29399d43093cee92a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e97d719a7c094de29399d43093cee92a. Latest consensus state: current_term: 1 leader_uuid: "e97d719a7c094de29399d43093cee92a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e97d719a7c094de29399d43093cee92a" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 45425 } } }
I20260928 06:37:41.013504 6315 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:37:41.013599 6308 sys_catalog.cc:458] T 00000000000000000000000000000000 P e97d719a7c094de29399d43093cee92a [sys.catalog]: This master's current role is: LEADER
I20260928 06:37:41.014218 6315 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:37:41.017073 6315 catalog_manager.cc:1347] Generated new cluster ID: 29199c858ae84fa79abbc602d21fce93
I20260928 06:37:41.017161 6315 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:37:41.019385 6322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:41.013858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35743 (local address 127.16.157.126:45425)
0928 06:37:41.014250 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:41.014259 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:41.014291 (+ 32us) server_negotiation.cc:407] Connection header received
0928 06:37:41.014423 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:41.014430 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:41.014497 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:41.014636 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:41.014867 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:41.016004 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:41.017121 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:41.017879 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:41.018005 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:41.018689 (+ 684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:41.018693 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:41.018694 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:37:41.018911 (+ 217us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:41.018950 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:37:41.018965 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:41.019060 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:41.019140 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 06:37:41.019167 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":88,"threads_started":1}
I20260928 06:37:41.020273 17013 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:37:41.022531 6315 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:37:41.022984 6315 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:37:41.027757 6315 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e97d719a7c094de29399d43093cee92a: Generated new TSK 0
I20260928 06:37:41.027911 6315 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:37:41.028753 6322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:41.021929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59303 (local address 127.16.157.126:45425)
0928 06:37:41.021971 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:41.021973 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:41.022284 (+ 311us) server_negotiation.cc:407] Connection header received
0928 06:37:41.022355 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:41.022356 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:41.022367 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:41.022413 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:41.025760 (+ 3347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:41.026526 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:41.027548 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:41.027865 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:41.027923 (+ 58us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:41.028545 (+ 622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:41.028546 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:41.028546 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:37:41.028550 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:41.028563 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:37:41.028568 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:41.028629 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:41.028676 (+ 47us) server_negotiation.cc:299] Negotiation successful
0928 06:37:41.028687 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:37:41.030678 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 6235
W20260928 06:37:41.034771 6220 connection.cc:535] client connection to 127.16.157.126:45425 recv error: Network error: failed to read from TLS socket (remote: 127.16.157.126:45425): Connection reset by peer (error 104)
2026-09-28T06:37:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1290 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577196946045-17013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:37:41 dist-test-slave-rrj5 krb5kdc[6334](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:37:41 dist-test-slave-rrj5 krb5kdc[6334](info): set up 2 sockets
Sep 28 06:37:41 dist-test-slave-rrj5 krb5kdc[6334](info): commencing operation
krb5kdc: starting...
W20260928 06:37:42.054466 17013 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Sep 28 06:37:42 dist-test-slave-rrj5 krb5kdc[6334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577462, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:37:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:37:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.157.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab.
Entry for principal kudu/127.16.157.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab.
I20260928 06:37:42.173966 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577196946045-17013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577196946045-17013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577196946045-17013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:33445
--webserver_interface=127.16.157.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=kudu/127.16.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:34821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.157.126:33445
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:37:42.252591 6347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:42.252780 6347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:42.252802 6347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:42.253888 6347 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:37:42.253923 6347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:42.253937 6347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:42.253950 6347 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:37:42.253962 6347 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:37:42.255330 6347 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:34821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577196946045-17013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577196946045-17013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.157.126:33445
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=kudu/127.16.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:33445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577196946045-17013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:37:42.255518 6347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:42.255681 6347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:42.260360 6353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:42.260370 6352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:42.260448 6347 server_base.cc:1034] running on GCE node
W20260928 06:37:42.260387 6355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:42.260828 6347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:42.261108 6347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:42.262264 6347 hybrid_clock.cc:648] HybridClock initialized: now 1790577462262255 us; error 34 us; skew 500 ppm
Sep 28 06:37:42 dist-test-slave-rrj5 krb5kdc[6334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577462, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:42.264709 6347 init.cc:377] Logged in from keytab as kudu/127.16.157.126@KRBTEST.COM (short username kudu)
I20260928 06:37:42.265168 6347 webserver.cc:460] Webserver started at http://127.16.157.126:41585/ using document root <none> and password file <none>
I20260928 06:37:42.265403 6347 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:42.265447 6347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:42.265643 6347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:37:42.266690 6347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577196946045-17013-0/minicluster-data/master-0/data/instance:
uuid: "5544a7f9245b486e84b66c98639217c5"
format_stamp: "Formatted at 2026-09-28 06:37:42 on dist-test-slave-rrj5"
server_key: "57f1ef644022233704032522c88c425b"
server_key_iv: "180b9378102996f59908b69eb2927f01"
server_key_version: "encryptionkey@0"
I20260928 06:37:42.267030 6347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577196946045-17013-0/minicluster-data/master-0/wal/instance:
uuid: "5544a7f9245b486e84b66c98639217c5"
format_stamp: "Formatted at 2026-09-28 06:37:42 on dist-test-slave-rrj5"
server_key: "57f1ef644022233704032522c88c425b"
server_key_iv: "180b9378102996f59908b69eb2927f01"
server_key_version: "encryptionkey@0"
I20260928 06:37:42.268308 6347 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:37:42.269055 6362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:42.269236 6347 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:37:42.269322 6347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577196946045-17013-0/minicluster-data/master-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577196946045-17013-0/minicluster-data/master-0/wal
uuid: "5544a7f9245b486e84b66c98639217c5"
format_stamp: "Formatted at 2026-09-28 06:37:42 on dist-test-slave-rrj5"
server_key: "57f1ef644022233704032522c88c425b"
server_key_iv: "180b9378102996f59908b69eb2927f01"
server_key_version: "encryptionkey@0"
I20260928 06:37:42.269375 6347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577196946045-17013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577196946045-17013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577196946045-17013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:42.272475 6347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:42.274250 6347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:42.274353 6347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:42.278070 6347 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.126:33445
I20260928 06:37:42.278115 6414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.126:33445 every 8 connection(s)
I20260928 06:37:42.278429 6347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
I20260928 06:37:42.279356 6415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:37:42.281443 6415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5544a7f9245b486e84b66c98639217c5: Bootstrap starting.
I20260928 06:37:42.282032 6415 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5544a7f9245b486e84b66c98639217c5: Neither blocks nor log segments found. Creating new log.
I20260928 06:37:42.282317 6415 log.cc:824] T 00000000000000000000000000000000 P 5544a7f9245b486e84b66c98639217c5: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:42.282966 6415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5544a7f9245b486e84b66c98639217c5: No bootstrap required, opened a new log
I20260928 06:37:42.284520 6415 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5544a7f9245b486e84b66c98639217c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5544a7f9245b486e84b66c98639217c5" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 33445 } }
I20260928 06:37:42.284672 6415 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5544a7f9245b486e84b66c98639217c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:37:42.284718 6415 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5544a7f9245b486e84b66c98639217c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5544a7f9245b486e84b66c98639217c5, State: Initialized, Role: FOLLOWER
I20260928 06:37:42.284870 6415 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5544a7f9245b486e84b66c98639217c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5544a7f9245b486e84b66c98639217c5" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 33445 } }
I20260928 06:37:42.284950 6415 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5544a7f9245b486e84b66c98639217c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:37:42.285001 6415 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5544a7f9245b486e84b66c98639217c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:37:42.285055 6415 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5544a7f9245b486e84b66c98639217c5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:37:42.285882 6415 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5544a7f9245b486e84b66c98639217c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5544a7f9245b486e84b66c98639217c5" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 33445 } }
I20260928 06:37:42.286000 6415 leader_election.cc:302] T 00000000000000000000000000000000 P 5544a7f9245b486e84b66c98639217c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5544a7f9245b486e84b66c98639217c5; no voters:
I20260928 06:37:42.286181 6415 leader_election.cc:288] T 00000000000000000000000000000000 P 5544a7f9245b486e84b66c98639217c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:37:42.286219 6418 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5544a7f9245b486e84b66c98639217c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:37:42.286377 6418 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5544a7f9245b486e84b66c98639217c5 [term 1 LEADER]: Becoming Leader. State: Replica: 5544a7f9245b486e84b66c98639217c5, State: Running, Role: LEADER
I20260928 06:37:42.286432 6415 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5544a7f9245b486e84b66c98639217c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:37:42.286549 6418 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5544a7f9245b486e84b66c98639217c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5544a7f9245b486e84b66c98639217c5" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 33445 } }
I20260928 06:37:42.286906 6419 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5544a7f9245b486e84b66c98639217c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5544a7f9245b486e84b66c98639217c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5544a7f9245b486e84b66c98639217c5" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 33445 } } }
I20260928 06:37:42.287050 6419 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5544a7f9245b486e84b66c98639217c5 [sys.catalog]: This master's current role is: LEADER
I20260928 06:37:42.286919 6420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5544a7f9245b486e84b66c98639217c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5544a7f9245b486e84b66c98639217c5. Latest consensus state: current_term: 1 leader_uuid: "5544a7f9245b486e84b66c98639217c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5544a7f9245b486e84b66c98639217c5" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 33445 } } }
I20260928 06:37:42.287161 6420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5544a7f9245b486e84b66c98639217c5 [sys.catalog]: This master's current role is: LEADER
I20260928 06:37:42.287531 6427 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:37:42.288026 6427 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:37:42.288035 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 6347
I20260928 06:37:42.288134 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577196946045-17013-0/minicluster-data/master-0/wal/instance
I20260928 06:37:42.288335 17013 external_mini_cluster.cc:1428] Setting key 7ddbc54e6a08091d2e290f08e2a66871
I20260928 06:37:42.289707 6427 catalog_manager.cc:1347] Generated new cluster ID: 1b84995d51a84e59a2d183595ba75f26
I20260928 06:37:42.289757 6427 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:37:42.294613 6437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:42.288810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44957 (local address 127.16.157.126:33445)
0928 06:37:42.288980 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:42.288989 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:42.289173 (+ 184us) server_negotiation.cc:407] Connection header received
0928 06:37:42.289333 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:42.289340 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:42.289371 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:42.289517 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:42.294185 (+ 4668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:42.294188 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:42.294189 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:37:42.294193 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:42.294232 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:37:42.294245 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:42.294308 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:42.294413 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 06:37:42.294451 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":84,"threads_started":1}
I20260928 06:37:42.295461 17013 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:37:42.298977 6427 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:37:42.299126 6427 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:37:42.299832 6437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:42.297917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38947 (local address 127.16.157.126:33445)
0928 06:37:42.297957 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:42.297959 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:42.298687 (+ 728us) server_negotiation.cc:407] Connection header received
0928 06:37:42.298889 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:42.298890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:42.298896 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:42.298956 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:42.299600 (+ 644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:42.299601 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:42.299601 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:37:42.299606 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:42.299620 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:37:42.299624 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:42.299655 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:42.299760 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 06:37:42.299773 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 06:37:42.305029 6427 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5544a7f9245b486e84b66c98639217c5: Generated new TSK 0
I20260928 06:37:42.305197 6427 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:37:42.309592 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 6347
2026-09-28T06:37:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1279 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577196946045-17013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:37:42 dist-test-slave-rrj5 krb5kdc[6446](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:37:42 dist-test-slave-rrj5 krb5kdc[6446](info): set up 2 sockets
Sep 28 06:37:42 dist-test-slave-rrj5 krb5kdc[6446](info): commencing operation
krb5kdc: starting...
W20260928 06:37:43.334342 17013 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Sep 28 06:37:43 dist-test-slave-rrj5 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577463, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:37:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:37:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.157.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab.
Entry for principal kudu/127.16.157.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab.
I20260928 06:37:43.452118 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577196946045-17013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577196946045-17013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577196946045-17013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:34003
--webserver_interface=127.16.157.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=kudu/127.16.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:39921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.157.126:34003
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:37:43.530378 6459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:43.530615 6459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:43.530652 6459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:43.531777 6459 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:37:43.531844 6459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:43.531869 6459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:43.531899 6459 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:37:43.531920 6459 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:37:43.533356 6459 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:39921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577196946045-17013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577196946045-17013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.157.126:34003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=kudu/127.16.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:34003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577196946045-17013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:37:43.533650 6459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:43.533870 6459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:43.538512 6464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:43.538524 6465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:43.538524 6467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:43.538756 6459 server_base.cc:1034] running on GCE node
I20260928 06:37:43.538987 6459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:43.539278 6459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:43.540467 6459 hybrid_clock.cc:648] HybridClock initialized: now 1790577463540434 us; error 35 us; skew 500 ppm
Sep 28 06:37:43 dist-test-slave-rrj5 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577463, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:43.542773 6459 init.cc:377] Logged in from keytab as kudu/127.16.157.126@KRBTEST.COM (short username kudu)
I20260928 06:37:43.543187 6459 webserver.cc:460] Webserver started at http://127.16.157.126:46169/ using document root <none> and password file <none>
I20260928 06:37:43.543377 6459 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:43.543416 6459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:43.543503 6459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:37:43.544409 6459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577196946045-17013-0/minicluster-data/master-0/data/instance:
uuid: "98e41a41076145c2b7e5a578f5d51c91"
format_stamp: "Formatted at 2026-09-28 06:37:43 on dist-test-slave-rrj5"
server_key: "ab74ba48a4f7baa3805194a214aa7602"
server_key_iv: "a0d908ad0b8918b3ae82adfc56747d77"
server_key_version: "encryptionkey@0"
I20260928 06:37:43.544708 6459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577196946045-17013-0/minicluster-data/master-0/wal/instance:
uuid: "98e41a41076145c2b7e5a578f5d51c91"
format_stamp: "Formatted at 2026-09-28 06:37:43 on dist-test-slave-rrj5"
server_key: "ab74ba48a4f7baa3805194a214aa7602"
server_key_iv: "a0d908ad0b8918b3ae82adfc56747d77"
server_key_version: "encryptionkey@0"
I20260928 06:37:43.546051 6459 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:37:43.546736 6474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:43.546904 6459 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:37:43.546976 6459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577196946045-17013-0/minicluster-data/master-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577196946045-17013-0/minicluster-data/master-0/wal
uuid: "98e41a41076145c2b7e5a578f5d51c91"
format_stamp: "Formatted at 2026-09-28 06:37:43 on dist-test-slave-rrj5"
server_key: "ab74ba48a4f7baa3805194a214aa7602"
server_key_iv: "a0d908ad0b8918b3ae82adfc56747d77"
server_key_version: "encryptionkey@0"
I20260928 06:37:43.547029 6459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577196946045-17013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577196946045-17013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577196946045-17013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:43.573801 6459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:43.575757 6459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:43.575870 6459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:43.580034 6459 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.126:34003
I20260928 06:37:43.580099 6526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.126:34003 every 8 connection(s)
I20260928 06:37:43.580432 6459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
I20260928 06:37:43.581297 6527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:37:43.583532 6527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98e41a41076145c2b7e5a578f5d51c91: Bootstrap starting.
I20260928 06:37:43.584040 6527 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98e41a41076145c2b7e5a578f5d51c91: Neither blocks nor log segments found. Creating new log.
I20260928 06:37:43.584286 6527 log.cc:824] T 00000000000000000000000000000000 P 98e41a41076145c2b7e5a578f5d51c91: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:43.584919 6527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98e41a41076145c2b7e5a578f5d51c91: No bootstrap required, opened a new log
I20260928 06:37:43.586432 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 6459
I20260928 06:37:43.586527 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577196946045-17013-0/minicluster-data/master-0/wal/instance
I20260928 06:37:43.586722 17013 external_mini_cluster.cc:1428] Setting key 815e90628edd9089aa7bbe883e805c28
I20260928 06:37:43.586688 6527 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98e41a41076145c2b7e5a578f5d51c91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98e41a41076145c2b7e5a578f5d51c91" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 34003 } }
I20260928 06:37:43.586871 6527 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98e41a41076145c2b7e5a578f5d51c91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:37:43.586921 6527 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98e41a41076145c2b7e5a578f5d51c91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98e41a41076145c2b7e5a578f5d51c91, State: Initialized, Role: FOLLOWER
I20260928 06:37:43.587059 6527 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98e41a41076145c2b7e5a578f5d51c91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98e41a41076145c2b7e5a578f5d51c91" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 34003 } }
I20260928 06:37:43.587143 6527 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98e41a41076145c2b7e5a578f5d51c91 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:37:43.587175 6527 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98e41a41076145c2b7e5a578f5d51c91 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:37:43.587278 6527 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98e41a41076145c2b7e5a578f5d51c91 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:37:43.588322 6527 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98e41a41076145c2b7e5a578f5d51c91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98e41a41076145c2b7e5a578f5d51c91" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 34003 } }
I20260928 06:37:43.588428 6527 leader_election.cc:302] T 00000000000000000000000000000000 P 98e41a41076145c2b7e5a578f5d51c91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98e41a41076145c2b7e5a578f5d51c91; no voters:
I20260928 06:37:43.588575 6527 leader_election.cc:288] T 00000000000000000000000000000000 P 98e41a41076145c2b7e5a578f5d51c91 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:37:43.588639 6532 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98e41a41076145c2b7e5a578f5d51c91 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:37:43.588814 6527 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98e41a41076145c2b7e5a578f5d51c91 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:37:43.588821 6532 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98e41a41076145c2b7e5a578f5d51c91 [term 1 LEADER]: Becoming Leader. State: Replica: 98e41a41076145c2b7e5a578f5d51c91, State: Running, Role: LEADER
I20260928 06:37:43.589020 6532 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98e41a41076145c2b7e5a578f5d51c91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98e41a41076145c2b7e5a578f5d51c91" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 34003 } }
I20260928 06:37:43.589365 6533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98e41a41076145c2b7e5a578f5d51c91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "98e41a41076145c2b7e5a578f5d51c91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98e41a41076145c2b7e5a578f5d51c91" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 34003 } } }
I20260928 06:37:43.589448 6533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98e41a41076145c2b7e5a578f5d51c91 [sys.catalog]: This master's current role is: LEADER
I20260928 06:37:43.589674 6541 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:37:43.589720 6534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98e41a41076145c2b7e5a578f5d51c91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98e41a41076145c2b7e5a578f5d51c91. Latest consensus state: current_term: 1 leader_uuid: "98e41a41076145c2b7e5a578f5d51c91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98e41a41076145c2b7e5a578f5d51c91" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 34003 } } }
I20260928 06:37:43.589782 6534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98e41a41076145c2b7e5a578f5d51c91 [sys.catalog]: This master's current role is: LEADER
I20260928 06:37:43.590312 6541 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:37:43.591871 6541 catalog_manager.cc:1347] Generated new cluster ID: f0d6c65eb2b04f048d85b42b1abc2697
I20260928 06:37:43.591922 6541 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:37:43 dist-test-slave-rrj5 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577463, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:37:43.598250 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:43.587144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34435 (local address 127.16.157.126:34003)
0928 06:37:43.587339 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:43.587348 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:43.587381 (+ 33us) server_negotiation.cc:407] Connection header received
0928 06:37:43.587511 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:43.587518 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:43.587591 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:43.587777 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:43.588368 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:43.589801 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:43.590958 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:43.591586 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:43.591700 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:43.594599 (+ 2899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:43.594603 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:43.594604 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:43.594615 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:43.596576 (+ 1961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:43.596949 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:43.596952 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:43.596952 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:43.597069 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:43.597271 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:43.597272 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:43.597272 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:43.597622 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:43.597837 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:43.597918 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 06:37:43.598057 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":85,"threads_started":1}
I20260928 06:37:43.599267 17013 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:37:43.601753 6541 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:37:43.602211 6541 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:37:43.606328 6541 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98e41a41076145c2b7e5a578f5d51c91: Generated new TSK 0
I20260928 06:37:43.606492 6541 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:37:43.607722 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:43.600670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44477 (local address 127.16.157.126:34003)
0928 06:37:43.600704 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:43.600706 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:43.600762 (+ 56us) server_negotiation.cc:407] Connection header received
0928 06:37:43.600824 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:43.600824 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:43.600834 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:43.600887 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:43.601525 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:43.602313 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:43.603356 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:43.603678 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:43.603718 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:43.605113 (+ 1395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:43.605114 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:43.605114 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:43.605121 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:43.606709 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:43.607084 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:43.607084 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:43.607085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:43.607134 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:43.607314 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:43.607315 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:43.607315 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:43.607404 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:43.607484 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:43.607606 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 06:37:43.607670 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:37:43.609494 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 6459
2026-09-28T06:37:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1299 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577196946045-17013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:37:43 dist-test-slave-rrj5 krb5kdc[6558](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:37:43 dist-test-slave-rrj5 krb5kdc[6558](info): set up 2 sockets
Sep 28 06:37:43 dist-test-slave-rrj5 krb5kdc[6558](info): commencing operation
krb5kdc: starting...
W20260928 06:37:44.623080 17013 mini_kdc.cc:121] Time spent starting KDC: real 0.996s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Sep 28 06:37:44 dist-test-slave-rrj5 krb5kdc[6558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:37:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:37:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.157.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab.
Entry for principal kudu/127.16.157.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab.
I20260928 06:37:44.739838 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577196946045-17013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577196946045-17013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577196946045-17013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:33407
--webserver_interface=127.16.157.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=kudu/127.16.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:46643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.157.126:33407
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:37:44.819751 6571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:44.819959 6571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:44.819985 6571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:44.821178 6571 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:37:44.821233 6571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:44.821249 6571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:44.821264 6571 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:37:44.821278 6571 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:37:44.822806 6571 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:46643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577196946045-17013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577196946045-17013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.157.126:33407
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=kudu/127.16.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:33407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577196946045-17013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:37:44.823094 6571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:44.823325 6571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:44.828454 6579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:44.828508 6577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:44.828564 6576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:44.828655 6571 server_base.cc:1034] running on GCE node
I20260928 06:37:44.828904 6571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:44.829166 6571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:44.830302 6571 hybrid_clock.cc:648] HybridClock initialized: now 1790577464830286 us; error 31 us; skew 500 ppm
Sep 28 06:37:44 dist-test-slave-rrj5 krb5kdc[6558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577464, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:44.832698 6571 init.cc:377] Logged in from keytab as kudu/127.16.157.126@KRBTEST.COM (short username kudu)
I20260928 06:37:44.833106 6571 webserver.cc:460] Webserver started at http://127.16.157.126:36487/ using document root <none> and password file <none>
I20260928 06:37:44.833333 6571 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:44.833398 6571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:44.833554 6571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:37:44.834555 6571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577196946045-17013-0/minicluster-data/master-0/data/instance:
uuid: "5035bb2ce31d48bb990c42d983143194"
format_stamp: "Formatted at 2026-09-28 06:37:44 on dist-test-slave-rrj5"
server_key: "94f5265c59c7459b9ee68cf537d215c7"
server_key_iv: "23ee9deca06174d00d5298867c9eca81"
server_key_version: "encryptionkey@0"
I20260928 06:37:44.834842 6571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577196946045-17013-0/minicluster-data/master-0/wal/instance:
uuid: "5035bb2ce31d48bb990c42d983143194"
format_stamp: "Formatted at 2026-09-28 06:37:44 on dist-test-slave-rrj5"
server_key: "94f5265c59c7459b9ee68cf537d215c7"
server_key_iv: "23ee9deca06174d00d5298867c9eca81"
server_key_version: "encryptionkey@0"
I20260928 06:37:44.836058 6571 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:37:44.836719 6586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:44.836894 6571 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:37:44.836985 6571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577196946045-17013-0/minicluster-data/master-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577196946045-17013-0/minicluster-data/master-0/wal
uuid: "5035bb2ce31d48bb990c42d983143194"
format_stamp: "Formatted at 2026-09-28 06:37:44 on dist-test-slave-rrj5"
server_key: "94f5265c59c7459b9ee68cf537d215c7"
server_key_iv: "23ee9deca06174d00d5298867c9eca81"
server_key_version: "encryptionkey@0"
I20260928 06:37:44.837065 6571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577196946045-17013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577196946045-17013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577196946045-17013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:44.846727 6571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:44.848397 6571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:44.848546 6571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:44.852332 6571 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.126:33407
I20260928 06:37:44.852367 6638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.126:33407 every 8 connection(s)
I20260928 06:37:44.852706 6571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
I20260928 06:37:44.853513 6639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:37:44.853955 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 6571
I20260928 06:37:44.854077 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577196946045-17013-0/minicluster-data/master-0/wal/instance
I20260928 06:37:44.854259 17013 external_mini_cluster.cc:1428] Setting key bedf0c7673ed6fb1b4cca6df1df83fed
I20260928 06:37:44.856252 6639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5035bb2ce31d48bb990c42d983143194: Bootstrap starting.
I20260928 06:37:44.856904 6639 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5035bb2ce31d48bb990c42d983143194: Neither blocks nor log segments found. Creating new log.
I20260928 06:37:44.857218 6639 log.cc:824] T 00000000000000000000000000000000 P 5035bb2ce31d48bb990c42d983143194: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:44.857944 6639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5035bb2ce31d48bb990c42d983143194: No bootstrap required, opened a new log
I20260928 06:37:44.859761 6642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:44.854764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37161 (local address 127.16.157.126:33407)
0928 06:37:44.854927 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:44.854935 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:44.854958 (+ 23us) server_negotiation.cc:407] Connection header received
0928 06:37:44.855342 (+ 384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:44.855348 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:44.855391 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:44.855500 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:44.855732 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:44.856639 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:44.857743 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:44.858171 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:44.858251 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:44.858931 (+ 680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:44.858935 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:44.858935 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:37:44.859087 (+ 152us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:44.859117 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:37:44.859127 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:44.859177 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:44.859283 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 06:37:44.859350 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":83,"threads_started":1}
I20260928 06:37:44.859728 6639 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5035bb2ce31d48bb990c42d983143194 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5035bb2ce31d48bb990c42d983143194" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 33407 } }
I20260928 06:37:44.859894 6639 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5035bb2ce31d48bb990c42d983143194 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:37:44.859944 6639 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5035bb2ce31d48bb990c42d983143194 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5035bb2ce31d48bb990c42d983143194, State: Initialized, Role: FOLLOWER
I20260928 06:37:44.860074 6639 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5035bb2ce31d48bb990c42d983143194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5035bb2ce31d48bb990c42d983143194" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 33407 } }
I20260928 06:37:44.860133 6639 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5035bb2ce31d48bb990c42d983143194 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:37:44.860170 6639 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5035bb2ce31d48bb990c42d983143194 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:37:44.860219 6639 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5035bb2ce31d48bb990c42d983143194 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:37:44.860735 17013 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:37:44.860924 6639 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5035bb2ce31d48bb990c42d983143194 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5035bb2ce31d48bb990c42d983143194" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 33407 } }
I20260928 06:37:44.861042 6639 leader_election.cc:302] T 00000000000000000000000000000000 P 5035bb2ce31d48bb990c42d983143194 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5035bb2ce31d48bb990c42d983143194; no voters:
I20260928 06:37:44.861234 6639 leader_election.cc:288] T 00000000000000000000000000000000 P 5035bb2ce31d48bb990c42d983143194 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:37:44.861264 6644 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5035bb2ce31d48bb990c42d983143194 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:37:44.861371 6644 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5035bb2ce31d48bb990c42d983143194 [term 1 LEADER]: Becoming Leader. State: Replica: 5035bb2ce31d48bb990c42d983143194, State: Running, Role: LEADER
I20260928 06:37:44.861496 6639 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5035bb2ce31d48bb990c42d983143194 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:37:44.861506 6644 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5035bb2ce31d48bb990c42d983143194 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5035bb2ce31d48bb990c42d983143194" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 33407 } }
I20260928 06:37:44.861896 6644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5035bb2ce31d48bb990c42d983143194 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5035bb2ce31d48bb990c42d983143194. Latest consensus state: current_term: 1 leader_uuid: "5035bb2ce31d48bb990c42d983143194" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5035bb2ce31d48bb990c42d983143194" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 33407 } } }
I20260928 06:37:44.861908 6645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5035bb2ce31d48bb990c42d983143194 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5035bb2ce31d48bb990c42d983143194" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5035bb2ce31d48bb990c42d983143194" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 33407 } } }
I20260928 06:37:44.862010 6644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5035bb2ce31d48bb990c42d983143194 [sys.catalog]: This master's current role is: LEADER
I20260928 06:37:44.862020 6645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5035bb2ce31d48bb990c42d983143194 [sys.catalog]: This master's current role is: LEADER
I20260928 06:37:44.862226 6655 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:37:44.862691 6655 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:37:44.864322 6655 catalog_manager.cc:1347] Generated new cluster ID: eef01aea6e574a60abcce830a78907e7
I20260928 06:37:44.864380 6655 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:37:44.866792 6642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:44.863146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59749 (local address 127.16.157.126:33407)
0928 06:37:44.863184 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:44.863186 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:44.863414 (+ 228us) server_negotiation.cc:407] Connection header received
0928 06:37:44.863480 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:44.863481 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:44.863491 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:44.863549 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:44.863700 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:44.864373 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:44.865398 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:44.865702 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:44.865764 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:44.866477 (+ 713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:44.866479 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:44.866479 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:37:44.866483 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:44.866497 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:37:44.866502 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:44.866547 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:44.866704 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 06:37:44.866724 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:37:44.885378 6655 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:37:44.886168 6655 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:37:44.909765 6655 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5035bb2ce31d48bb990c42d983143194: Generated new TSK 0
I20260928 06:37:44.909994 6655 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:37:44.921940 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 6571
2026-09-28T06:37:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1312 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577196946045-17013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:37:44 dist-test-slave-rrj5 krb5kdc[6670](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:37:44 dist-test-slave-rrj5 krb5kdc[6670](info): set up 2 sockets
Sep 28 06:37:44 dist-test-slave-rrj5 krb5kdc[6670](info): commencing operation
krb5kdc: starting...
W20260928 06:37:45.946683 17013 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Sep 28 06:37:46 dist-test-slave-rrj5 krb5kdc[6670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:37:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:37:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.157.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab.
Entry for principal kudu/127.16.157.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab.
I20260928 06:37:46.063975 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577196946045-17013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577196946045-17013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577196946045-17013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:44193
--webserver_interface=127.16.157.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=kudu/127.16.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:35261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.157.126:44193
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:37:46.143427 6683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:46.143612 6683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:46.143635 6683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:46.144701 6683 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:37:46.144735 6683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:46.144750 6683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:46.144762 6683 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:37:46.144774 6683 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:37:46.146179 6683 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:35261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577196946045-17013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577196946045-17013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.157.126:44193
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=kudu/127.16.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:44193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577196946045-17013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:37:46.146358 6683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:46.146519 6683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:46.151176 6689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:46.151203 6688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:46.151177 6691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:46.151419 6683 server_base.cc:1034] running on GCE node
I20260928 06:37:46.151654 6683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:46.151890 6683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:46.153070 6683 hybrid_clock.cc:648] HybridClock initialized: now 1790577466153061 us; error 27 us; skew 500 ppm
Sep 28 06:37:46 dist-test-slave-rrj5 krb5kdc[6670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577466, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:46.158092 6683 init.cc:377] Logged in from keytab as kudu/127.16.157.126@KRBTEST.COM (short username kudu)
I20260928 06:37:46.158522 6683 webserver.cc:460] Webserver started at http://127.16.157.126:44217/ using document root <none> and password file <none>
I20260928 06:37:46.158694 6683 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:46.158731 6683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:46.158814 6683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:37:46.159791 6683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577196946045-17013-0/minicluster-data/master-0/data/instance:
uuid: "d1e321de6ccd40718ae965e88a0b9419"
format_stamp: "Formatted at 2026-09-28 06:37:46 on dist-test-slave-rrj5"
server_key: "c60d51d12c2ddedf8d937f9c9c281e9a"
server_key_iv: "eb9963d281644fdb72fa355009761f52"
server_key_version: "encryptionkey@0"
I20260928 06:37:46.160130 6683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577196946045-17013-0/minicluster-data/master-0/wal/instance:
uuid: "d1e321de6ccd40718ae965e88a0b9419"
format_stamp: "Formatted at 2026-09-28 06:37:46 on dist-test-slave-rrj5"
server_key: "c60d51d12c2ddedf8d937f9c9c281e9a"
server_key_iv: "eb9963d281644fdb72fa355009761f52"
server_key_version: "encryptionkey@0"
I20260928 06:37:46.161357 6683 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 06:37:46.162088 6698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:46.162251 6683 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:46.162346 6683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577196946045-17013-0/minicluster-data/master-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577196946045-17013-0/minicluster-data/master-0/wal
uuid: "d1e321de6ccd40718ae965e88a0b9419"
format_stamp: "Formatted at 2026-09-28 06:37:46 on dist-test-slave-rrj5"
server_key: "c60d51d12c2ddedf8d937f9c9c281e9a"
server_key_iv: "eb9963d281644fdb72fa355009761f52"
server_key_version: "encryptionkey@0"
I20260928 06:37:46.162427 6683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577196946045-17013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577196946045-17013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577196946045-17013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:46.165684 6683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:46.167362 6683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:46.167512 6683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:46.171579 6683 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.126:44193
I20260928 06:37:46.171607 6750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.126:44193 every 8 connection(s)
I20260928 06:37:46.171937 6683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
I20260928 06:37:46.172823 6751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:37:46.174985 6751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1e321de6ccd40718ae965e88a0b9419: Bootstrap starting.
I20260928 06:37:46.175485 6751 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1e321de6ccd40718ae965e88a0b9419: Neither blocks nor log segments found. Creating new log.
I20260928 06:37:46.175710 6751 log.cc:824] T 00000000000000000000000000000000 P d1e321de6ccd40718ae965e88a0b9419: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:46.176290 6751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1e321de6ccd40718ae965e88a0b9419: No bootstrap required, opened a new log
I20260928 06:37:46.177953 6751 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1e321de6ccd40718ae965e88a0b9419 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1e321de6ccd40718ae965e88a0b9419" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 44193 } }
I20260928 06:37:46.178076 6751 raft_consensus.cc:374] T 00000000000000000000000000000000 P d1e321de6ccd40718ae965e88a0b9419 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:37:46.178097 6751 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1e321de6ccd40718ae965e88a0b9419 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e321de6ccd40718ae965e88a0b9419, State: Initialized, Role: FOLLOWER
I20260928 06:37:46.178185 6751 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1e321de6ccd40718ae965e88a0b9419 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1e321de6ccd40718ae965e88a0b9419" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 44193 } }
I20260928 06:37:46.178231 6751 raft_consensus.cc:388] T 00000000000000000000000000000000 P d1e321de6ccd40718ae965e88a0b9419 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:37:46.178251 6751 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1e321de6ccd40718ae965e88a0b9419 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:37:46.178279 6751 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1e321de6ccd40718ae965e88a0b9419 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:37:46.178599 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 6683
I20260928 06:37:46.178712 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577196946045-17013-0/minicluster-data/master-0/wal/instance
I20260928 06:37:46.178874 17013 external_mini_cluster.cc:1428] Setting key ec277bfb0607f4f5a7b955b6b60234b0
I20260928 06:37:46.179097 6751 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1e321de6ccd40718ae965e88a0b9419 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1e321de6ccd40718ae965e88a0b9419" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 44193 } }
I20260928 06:37:46.179212 6751 leader_election.cc:302] T 00000000000000000000000000000000 P d1e321de6ccd40718ae965e88a0b9419 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1e321de6ccd40718ae965e88a0b9419; no voters:
I20260928 06:37:46.179411 6751 leader_election.cc:288] T 00000000000000000000000000000000 P d1e321de6ccd40718ae965e88a0b9419 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:37:46.179571 6757 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1e321de6ccd40718ae965e88a0b9419 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:37:46.179666 6751 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1e321de6ccd40718ae965e88a0b9419 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:37:46.179687 6757 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1e321de6ccd40718ae965e88a0b9419 [term 1 LEADER]: Becoming Leader. State: Replica: d1e321de6ccd40718ae965e88a0b9419, State: Running, Role: LEADER
I20260928 06:37:46.179768 6757 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1e321de6ccd40718ae965e88a0b9419 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1e321de6ccd40718ae965e88a0b9419" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 44193 } }
I20260928 06:37:46.180112 6757 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1e321de6ccd40718ae965e88a0b9419 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1e321de6ccd40718ae965e88a0b9419. Latest consensus state: current_term: 1 leader_uuid: "d1e321de6ccd40718ae965e88a0b9419" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1e321de6ccd40718ae965e88a0b9419" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 44193 } } }
I20260928 06:37:46.180126 6758 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1e321de6ccd40718ae965e88a0b9419 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d1e321de6ccd40718ae965e88a0b9419" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1e321de6ccd40718ae965e88a0b9419" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 44193 } } }
I20260928 06:37:46.180210 6757 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1e321de6ccd40718ae965e88a0b9419 [sys.catalog]: This master's current role is: LEADER
I20260928 06:37:46.180234 6758 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1e321de6ccd40718ae965e88a0b9419 [sys.catalog]: This master's current role is: LEADER
I20260928 06:37:46.181121 6763 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:37:46.181397 6756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:46.179375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59551 (local address 127.16.157.126:44193)
0928 06:37:46.179677 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:46.179687 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:46.179720 (+ 33us) server_negotiation.cc:407] Connection header received
0928 06:37:46.179864 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:46.179871 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:46.179900 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:46.180302 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:46.180735 (+ 433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:46.180738 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:46.180739 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:37:46.180743 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:46.180783 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:37:46.180802 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:46.180881 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:46.181014 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 06:37:46.181057 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":94,"threads_started":1}
I20260928 06:37:46.181699 6763 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:37:46.182437 17013 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:37:46.183348 6763 catalog_manager.cc:1347] Generated new cluster ID: 48d8da9c71834eefa0cbb663d247a2a3
I20260928 06:37:46.183403 6763 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:37:46.186704 6756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:46.183965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37481 (local address 127.16.157.126:44193)
0928 06:37:46.184001 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:46.184003 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:46.185762 (+ 1759us) server_negotiation.cc:407] Connection header received
0928 06:37:46.185834 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:46.185836 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:46.185851 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:46.185899 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:46.186477 (+ 578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:46.186478 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:46.186478 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:37:46.186482 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:46.186494 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:37:46.186498 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:46.186544 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:46.186598 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 06:37:46.186612 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:37:46.196792 6763 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:37:46.196938 6763 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:37:46.204713 6763 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d1e321de6ccd40718ae965e88a0b9419: Generated new TSK 0
I20260928 06:37:46.204878 6763 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:37:46.207391 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 6683
2026-09-28T06:37:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1285 ms)
[----------] 18 tests from AuthTokenIssuingTest (23553 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:37:46 dist-test-slave-rrj5 krb5kdc[6782](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:37:46 dist-test-slave-rrj5 krb5kdc[6782](info): set up 2 sockets
Sep 28 06:37:46 dist-test-slave-rrj5 krb5kdc[6782](info): commencing operation
krb5kdc: starting...
W20260928 06:37:47.238467 17013 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Sep 28 06:37:47 dist-test-slave-rrj5 krb5kdc[6782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:37:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:37:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.157.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab.
Entry for principal kudu/127.16.157.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab.
I20260928 06:37:47.354458 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:45035
--webserver_interface=127.16.157.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=kudu/127.16.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:37317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.157.126:45035,127.16.157.125:41097,127.16.157.124:44225
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:37:47.434312 6795 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 06:37:47.434527 6795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:47.434545 6795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:47.434594 6795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:47.435729 6795 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:37:47.435770 6795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:47.435784 6795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:47.435837 6795 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:37:47.435854 6795 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:37:47.437294 6795 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:37317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.157.126:45035,127.16.157.125:41097,127.16.157.124:44225
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=kudu/127.16.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:45035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:37:47.437541 6795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:47.437785 6795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:47.442602 6801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:47.442613 6803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:47.442675 6795 server_base.cc:1034] running on GCE node
W20260928 06:37:47.442613 6800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:47.443074 6795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:47.443369 6795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:47.444533 6795 hybrid_clock.cc:648] HybridClock initialized: now 1790577467444523 us; error 64 us; skew 500 ppm
Sep 28 06:37:47 dist-test-slave-rrj5 krb5kdc[6782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577467, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:47.446941 6795 init.cc:377] Logged in from keytab as kudu/127.16.157.126@KRBTEST.COM (short username kudu)
I20260928 06:37:47.447358 6795 webserver.cc:460] Webserver started at http://127.16.157.126:41325/ using document root <none> and password file <none>
I20260928 06:37:47.447574 6795 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:47.447639 6795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:47.447788 6795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:37:47.448804 6795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/minicluster-data/master-0/data/instance:
uuid: "955d82548c254fcb994c27b3628f50dd"
format_stamp: "Formatted at 2026-09-28 06:37:47 on dist-test-slave-rrj5"
server_key: "4cd6bd13526ef7150f03d46f519e06b6"
server_key_iv: "e6f3de54dcefac3ae27276b95543e49f"
server_key_version: "encryptionkey@0"
I20260928 06:37:47.449164 6795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/minicluster-data/master-0/wal/instance:
uuid: "955d82548c254fcb994c27b3628f50dd"
format_stamp: "Formatted at 2026-09-28 06:37:47 on dist-test-slave-rrj5"
server_key: "4cd6bd13526ef7150f03d46f519e06b6"
server_key_iv: "e6f3de54dcefac3ae27276b95543e49f"
server_key_version: "encryptionkey@0"
I20260928 06:37:47.450519 6795 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 06:37:47.451220 6810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:47.451396 6795 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:47.451531 6795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/minicluster-data/master-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/minicluster-data/master-0/wal
uuid: "955d82548c254fcb994c27b3628f50dd"
format_stamp: "Formatted at 2026-09-28 06:37:47 on dist-test-slave-rrj5"
server_key: "4cd6bd13526ef7150f03d46f519e06b6"
server_key_iv: "e6f3de54dcefac3ae27276b95543e49f"
server_key_version: "encryptionkey@0"
I20260928 06:37:47.451602 6795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:47.474529 6795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:47.476342 6795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:47.476502 6795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:47.480485 6795 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.126:45035
I20260928 06:37:47.480545 6862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.126:45035 every 8 connection(s)
I20260928 06:37:47.480898 6795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
I20260928 06:37:47.481869 6863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:37:47.483434 6863 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.157.126" port: 45035 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:37:47.488983 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 6795
Sep 28 06:37:47 dist-test-slave-rrj5 krb5kdc[6782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577467, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.126@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:37:47.489116 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/minicluster-data/master-0/wal/instance
I20260928 06:37:47.489311 17013 external_mini_cluster.cc:1428] Setting key 66fc97397844dd3f2529fe457bb42c9c
I20260928 06:37:47.492561 6865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:47.484092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:45035 (local address 127.0.0.1:50468)
0928 06:37:47.484262 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:47.484308 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:47.484777 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:47.485035 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:47.485037 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:47.485275 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:37:47.485426 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:47.485430 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:47.486458 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:47.486459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:47.487112 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:47.487114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:47.487544 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:47.487545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:47.487599 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:47.488076 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:37:47.488084 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:37:47.489479 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:37:47.491104 (+ 1625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:47.491110 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:47.491112 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:47.491301 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:47.491557 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:47.491557 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:47.491558 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:47.491644 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:47.491992 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:37:47.491998 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:37:47.492056 (+ 58us) client_negotiation.cc:769] Sending connection context
0928 06:37:47.492158 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 06:37:47.492327 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":71,"threads_started":1}
I20260928 06:37:47.492570 6866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:47.484165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50468 (local address 127.16.157.126:45035)
0928 06:37:47.484333 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:47.484336 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:47.484401 (+ 65us) server_negotiation.cc:407] Connection header received
0928 06:37:47.484900 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:47.484902 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:47.484927 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:47.485001 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:47.485505 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:47.486379 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:47.487185 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:47.487472 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:47.487543 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:47.489561 (+ 2018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:47.489587 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:47.489587 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:47.489594 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:47.491005 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:47.491394 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:47.491397 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:47.491397 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:47.491477 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:47.491715 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:47.491716 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:47.491716 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:47.491920 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:47.492012 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:47.492205 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 06:37:47.492385 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":95,"server-negotiator.queue_time_us":139,"thread_start_us":103,"threads_started":1}
I20260928 06:37:47.493392 6863 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.157.125" port: 41097 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:37:47.493757 6865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:47.493639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.125:41097 (local address 127.0.0.1:46934)
0928 06:37:47.493688 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:47.493731 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.125:41097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 06:37:47.493887 6811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.157.125:41097: connect: Connection refused (error 111)
I20260928 06:37:47.494261 6865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:47.494171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.125:41097 (local address 127.0.0.1:46936)
0928 06:37:47.494215 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:47.494239 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.125:41097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 06:37:47.494371 6863 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.157.125:41097: Network error: Client connection negotiation failed: client connection to 127.16.157.125:41097: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.16.157.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.125.keytab.
Entry for principal kudu/127.16.157.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.125.keytab.
I20260928 06:37:47.512730 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.125:41097
--webserver_interface=127.16.157.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.125.keytab
--principal=kudu/127.16.157.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:37317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.157.126:45035,127.16.157.125:41097,127.16.157.124:44225
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:37:47.543746 6863 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.157.125" port: 41097 } attempt: 1
I20260928 06:37:47.544246 6865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:47.544111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.125:41097 (local address 127.0.0.1:46942)
0928 06:37:47.544182 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:47.544219 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.125:41097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 06:37:47.544615 6865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:47.544520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.125:41097 (local address 127.0.0.1:46952)
0928 06:37:47.544566 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:47.544592 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.125:41097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 06:37:47.544701 6863 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.157.125:41097: Network error: Client connection negotiation failed: client connection to 127.16.157.125:41097: connect: Connection refused (error 111)
W20260928 06:37:47.592564 6870 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 06:37:47.592829 6870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:47.592880 6870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:47.592943 6870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:47.594148 6870 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:37:47.594221 6870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:47.594245 6870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:47.594265 6870 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:37:47.594295 6870 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:37:47.595750 6870 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:37317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.157.126:45035,127.16.157.125:41097,127.16.157.124:44225
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.125.keytab
--principal=kudu/127.16.157.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.125:41097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.157.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:37:47.596048 6870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:47.596280 6870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:47.600967 6875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:47.601037 6878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:47.601058 6876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:47.601208 6870 server_base.cc:1034] running on GCE node
I20260928 06:37:47.601485 6870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:47.601783 6870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:47.602926 6870 hybrid_clock.cc:648] HybridClock initialized: now 1790577467602917 us; error 26 us; skew 500 ppm
Sep 28 06:37:47 dist-test-slave-rrj5 krb5kdc[6782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577467, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:47.605394 6870 init.cc:377] Logged in from keytab as kudu/127.16.157.125@KRBTEST.COM (short username kudu)
I20260928 06:37:47.605895 6870 webserver.cc:460] Webserver started at http://127.16.157.125:43727/ using document root <none> and password file <none>
I20260928 06:37:47.606148 6870 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:47.606194 6870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:47.606344 6870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:37:47.607347 6870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/minicluster-data/master-1/data/instance:
uuid: "149bb1d82393435bad11024bc557224c"
format_stamp: "Formatted at 2026-09-28 06:37:47 on dist-test-slave-rrj5"
server_key: "5c1ebed12e4ee917ac23a31d8bfead9f"
server_key_iv: "89fa3c89d6926153ad085c11056372c4"
server_key_version: "encryptionkey@0"
I20260928 06:37:47.607713 6870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/minicluster-data/master-1/wal/instance:
uuid: "149bb1d82393435bad11024bc557224c"
format_stamp: "Formatted at 2026-09-28 06:37:47 on dist-test-slave-rrj5"
server_key: "5c1ebed12e4ee917ac23a31d8bfead9f"
server_key_iv: "89fa3c89d6926153ad085c11056372c4"
server_key_version: "encryptionkey@0"
I20260928 06:37:47.609000 6870 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:37:47.609760 6885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:47.609932 6870 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:47.610026 6870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/minicluster-data/master-1/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/minicluster-data/master-1/wal
uuid: "149bb1d82393435bad11024bc557224c"
format_stamp: "Formatted at 2026-09-28 06:37:47 on dist-test-slave-rrj5"
server_key: "5c1ebed12e4ee917ac23a31d8bfead9f"
server_key_iv: "89fa3c89d6926153ad085c11056372c4"
server_key_version: "encryptionkey@0"
I20260928 06:37:47.610111 6870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:47.612834 6863 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.157.125" port: 41097 } attempt: 2
I20260928 06:37:47.613188 6865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:47.613065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.125:41097 (local address 127.0.0.1:46964)
0928 06:37:47.613121 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:47.613151 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.125:41097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 06:37:47.613564 6865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:47.613465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.125:41097 (local address 127.0.0.1:46978)
0928 06:37:47.613518 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:47.613542 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.125:41097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260928 06:37:47.613668 6863 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.157.125:41097: Network error: Client connection negotiation failed: client connection to 127.16.157.125:41097: connect: Connection refused (error 111)
I20260928 06:37:47.620239 6870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:47.622490 6870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:47.622651 6870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:47.626773 6870 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.125:41097
I20260928 06:37:47.626824 6937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.125:41097 every 8 connection(s)
I20260928 06:37:47.627166 6870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/minicluster-data/master-1/data/info.pb
I20260928 06:37:47.627959 6938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:37:47.629338 6938 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.157.126" port: 45035 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:37:47 dist-test-slave-rrj5 krb5kdc[6782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577467, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.125@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:37:47.637185 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 6870
I20260928 06:37:47.637288 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/minicluster-data/master-1/wal/instance
I20260928 06:37:47.637490 17013 external_mini_cluster.cc:1428] Setting key 763494fb0464c33d86098937a1d487b5
I20260928 06:37:47.637802 6866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:47.630016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50476 (local address 127.16.157.126:45035)
0928 06:37:47.630077 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:47.630080 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:47.630255 (+ 175us) server_negotiation.cc:407] Connection header received
0928 06:37:47.630759 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:47.630760 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:47.630774 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:47.630838 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:47.631404 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:47.631898 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:47.633034 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:47.633238 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:47.633320 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:47.635309 (+ 1989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:47.635310 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:47.635310 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:47.635317 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:47.636479 (+ 1162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:47.636841 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:47.636842 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:47.636842 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:47.636890 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:47.637133 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:47.637135 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:47.637135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:47.637257 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:47.637343 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:47.637674 (+ 331us) server_negotiation.cc:299] Negotiation successful
0928 06:37:47.637742 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 06:37:47.638072 6940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:47.629996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:45035 (local address 127.0.0.1:50476)
0928 06:37:47.630145 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:47.630176 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:47.630644 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:47.630909 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:47.630912 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:47.631126 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:37:47.631303 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:47.631307 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:47.631987 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:47.631988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:47.632951 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:47.632953 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:47.633335 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:47.633336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:47.633462 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:47.633922 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:37:47.633929 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:37:47.635225 (+ 1296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:37:47.636551 (+ 1326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:47.636557 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:47.636559 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:47.636767 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:47.636960 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:47.636961 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:47.636961 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:47.637054 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:47.637348 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:37:47.637356 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:37:47.637514 (+ 158us) client_negotiation.cc:769] Sending connection context
0928 06:37:47.637654 (+ 140us) client_negotiation.cc:240] Negotiation successful
0928 06:37:47.637817 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":66,"threads_started":1}
I20260928 06:37:47.638615 6938 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.157.125" port: 41097 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:37:47 dist-test-slave-rrj5 krb5kdc[6782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577467, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.125@KRBTEST.COM for kudu/127.16.157.125@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.157.124@KRBTEST.COM; defaulting to no policy
I20260928 06:37:47.646533 6940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:47.638847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.125:41097 (local address 127.0.0.1:46984)
0928 06:37:47.638888 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:47.638898 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:47.638932 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:47.639241 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:47.639243 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:47.639397 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:37:47.639465 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:47.639467 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:47.640272 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:47.640273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:47.641148 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:47.641152 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:47.641742 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:47.641744 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:47.641826 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:47.642161 (+ 335us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:37:47.642166 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:37:47.643437 (+ 1271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:37:47.645398 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:47.645399 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:47.645399 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:47.645565 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:47.645892 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:47.645893 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:47.645893 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:47.645953 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:47.646264 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:37:47.646264 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:37:47.646298 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 06:37:47.646348 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 06:37:47.646463 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 06:37:47.646700 6942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:47.638934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46984 (local address 127.16.157.125:41097)
0928 06:37:47.639088 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:47.639093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:47.639117 (+ 24us) server_negotiation.cc:407] Connection header received
0928 06:37:47.639138 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:47.639140 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:47.639151 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:47.639219 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:47.639560 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:47.640192 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:47.641230 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:47.641666 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:47.641720 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:47.643520 (+ 1800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:47.643528 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:47.643529 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:47.643536 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:47.645291 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:47.645740 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:47.645744 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:47.645744 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:47.645807 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:47.646021 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:47.646022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:47.646022 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:47.646195 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:47.646279 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:47.646387 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 06:37:47.646530 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":121,"thread_start_us":78,"threads_started":1}
I20260928 06:37:47.647177 6938 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.157.124" port: 44225 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:37:47.647544 6940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:47.647409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.124:44225 (local address 127.0.0.1:38812)
0928 06:37:47.647467 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:47.647513 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.124:44225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 06:37:47.647670 6889 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.157.124:44225: connect: Connection refused (error 111)
I20260928 06:37:47.648043 6940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:47.647953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.124:44225 (local address 127.0.0.1:38820)
0928 06:37:47.647999 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:47.648022 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.124:44225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 06:37:47.648160 6938 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.157.124:44225: Network error: Client connection negotiation failed: client connection to 127.16.157.124:44225: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.124@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.124.keytab.
Entry for principal kudu/127.16.157.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.124.keytab.
I20260928 06:37:47.662376 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.124:44225
--webserver_interface=127.16.157.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.124.keytab
--principal=kudu/127.16.157.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:37317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.157.126:45035,127.16.157.125:41097,127.16.157.124:44225
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:37:47.697379 6938 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.157.124" port: 44225 } attempt: 1
I20260928 06:37:47.697885 6940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:47.697734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.124:44225 (local address 127.0.0.1:38822)
0928 06:37:47.697821 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:47.697855 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.124:44225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 06:37:47.698238 6940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:47.698147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.124:44225 (local address 127.0.0.1:38836)
0928 06:37:47.698190 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:47.698215 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.124:44225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 06:37:47.698355 6938 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.157.124:44225: Network error: Client connection negotiation failed: client connection to 127.16.157.124:44225: connect: Connection refused (error 111)
I20260928 06:37:47.704807 6863 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.157.125" port: 41097 } attempt: 3
Sep 28 06:37:47 dist-test-slave-rrj5 krb5kdc[6782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577467, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.126@KRBTEST.COM for kudu/127.16.157.125@KRBTEST.COM
I20260928 06:37:47.716151 6865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:47.705136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.125:41097 (local address 127.0.0.1:47000)
0928 06:37:47.705198 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:47.705208 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:47.705254 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:47.705494 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:47.705495 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:47.705831 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:37:47.705949 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:47.705953 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:47.706944 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:47.706945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:47.707753 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:47.707756 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:47.708338 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:47.708340 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:47.708419 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:47.708847 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:37:47.708853 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:37:47.710298 (+ 1445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:37:47.714921 (+ 4623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:47.714922 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:47.714922 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:47.715193 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:47.715447 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:47.715448 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:47.715448 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:47.715512 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:47.715818 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:37:47.715820 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:37:47.715866 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 06:37:47.715953 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 06:37:47.716056 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 06:37:47.716149 6942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:47.705230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47000 (local address 127.16.157.125:41097)
0928 06:37:47.705309 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:47.705325 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:47.705332 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:37:47.705388 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:47.705389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:47.705401 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:47.705463 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:47.706044 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:47.706816 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:47.707879 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:47.708264 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:47.708336 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:47.710413 (+ 2077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:47.710415 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:47.710415 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:47.710423 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:47.714839 (+ 4416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:47.715276 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:47.715278 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:47.715278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:47.715358 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:47.715609 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:47.715610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:47.715610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:47.715738 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:47.715807 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:47.715980 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 06:37:47.716056 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 06:37:47.717063 6863 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.157.124" port: 44225 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:37:47.717399 6865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:47.717301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.124:44225 (local address 127.0.0.1:38846)
0928 06:37:47.717347 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:47.717374 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.124:44225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 06:37:47.717784 6865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:47.717691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.124:44225 (local address 127.0.0.1:38860)
0928 06:37:47.717741 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:47.717764 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.124:44225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 06:37:47.717900 6863 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.157.124:44225: Network error: Client connection negotiation failed: client connection to 127.16.157.124:44225: connect: Connection refused (error 111)
I20260928 06:37:47.749043 6863 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.157.124" port: 44225 } attempt: 1
I20260928 06:37:47.749506 6865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:47.749348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.124:44225 (local address 127.0.0.1:38870)
0928 06:37:47.749419 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:47.749465 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.124:44225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 06:37:47.749927 6865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:47.749815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.124:44225 (local address 127.0.0.1:38878)
0928 06:37:47.749867 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:47.749898 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.124:44225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 06:37:47.750015 6863 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.157.124:44225: Network error: Client connection negotiation failed: client connection to 127.16.157.124:44225: connect: Connection refused (error 111)
W20260928 06:37:47.758612 6945 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 06:37:47.758850 6945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:47.758911 6945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:47.758955 6945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:47.760099 6945 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:37:47.760161 6945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:47.760187 6945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:47.760213 6945 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:37:47.760241 6945 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:37:47.761693 6945 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:37317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.157.126:45035,127.16.157.125:41097,127.16.157.124:44225
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.124.keytab
--principal=kudu/127.16.157.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.124:44225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.157.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:37:47.761946 6945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:47.762149 6945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:37:47.766532 6938 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.157.124" port: 44225 } attempt: 2
W20260928 06:37:47.766721 6953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:47.766732 6951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:47.766973 6950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:47.767081 6940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:47.766930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.124:44225 (local address 127.0.0.1:38884)
0928 06:37:47.767002 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:47.767044 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.124:44225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 06:37:47.767125 6945 server_base.cc:1034] running on GCE node
I20260928 06:37:47.767354 6945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:47.767407 6940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:47.767319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.124:44225 (local address 127.0.0.1:38898)
0928 06:37:47.767363 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:47.767386 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.124:44225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 06:37:47.767504 6938 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.157.124:44225: Network error: Client connection negotiation failed: client connection to 127.16.157.124:44225: connect: Connection refused (error 111)
I20260928 06:37:47.767670 6945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:47.768836 6945 hybrid_clock.cc:648] HybridClock initialized: now 1790577467768824 us; error 25 us; skew 500 ppm
Sep 28 06:37:47 dist-test-slave-rrj5 krb5kdc[6782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577467, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:47.771368 6945 init.cc:377] Logged in from keytab as kudu/127.16.157.124@KRBTEST.COM (short username kudu)
I20260928 06:37:47.771847 6945 webserver.cc:460] Webserver started at http://127.16.157.124:34685/ using document root <none> and password file <none>
I20260928 06:37:47.772066 6945 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:47.772163 6945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:47.772327 6945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:37:47.773362 6945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/minicluster-data/master-2/data/instance:
uuid: "0795fc67e09240509506eceaac7fa988"
format_stamp: "Formatted at 2026-09-28 06:37:47 on dist-test-slave-rrj5"
server_key: "0332dab6161f8f41cf741b1f926740b3"
server_key_iv: "03c578b2d2cd59152dcbc1b12763298f"
server_key_version: "encryptionkey@0"
I20260928 06:37:47.773770 6945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/minicluster-data/master-2/wal/instance:
uuid: "0795fc67e09240509506eceaac7fa988"
format_stamp: "Formatted at 2026-09-28 06:37:47 on dist-test-slave-rrj5"
server_key: "0332dab6161f8f41cf741b1f926740b3"
server_key_iv: "03c578b2d2cd59152dcbc1b12763298f"
server_key_version: "encryptionkey@0"
I20260928 06:37:47.775079 6945 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:37:47.775871 6960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:47.776038 6945 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:37:47.776137 6945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/minicluster-data/master-2/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/minicluster-data/master-2/wal
uuid: "0795fc67e09240509506eceaac7fa988"
format_stamp: "Formatted at 2026-09-28 06:37:47 on dist-test-slave-rrj5"
server_key: "0332dab6161f8f41cf741b1f926740b3"
server_key_iv: "03c578b2d2cd59152dcbc1b12763298f"
server_key_version: "encryptionkey@0"
I20260928 06:37:47.776226 6945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:47.783393 6945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:47.785041 6945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:47.785195 6945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:47.789235 6945 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.124:44225
I20260928 06:37:47.789265 7012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.124:44225 every 8 connection(s)
I20260928 06:37:47.789688 6945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/minicluster-data/master-2/data/info.pb
I20260928 06:37:47.790464 7013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:37:47.791945 7013 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.157.126" port: 45035 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:37:47.796679 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 6945
I20260928 06:37:47.796778 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577196946045-17013-0/minicluster-data/master-2/wal/instance
I20260928 06:37:47.796945 17013 external_mini_cluster.cc:1428] Setting key 2918f09c3c35a56be55e3135b84d6a99
Sep 28 06:37:47 dist-test-slave-rrj5 krb5kdc[6782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577467, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.124@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:37:47.801131 6866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:47.792670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50480 (local address 127.16.157.126:45035)
0928 06:37:47.792743 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:47.792745 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:47.792950 (+ 205us) server_negotiation.cc:407] Connection header received
0928 06:37:47.793365 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:47.793366 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:47.793375 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:47.793418 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:47.794027 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:47.794490 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:47.795485 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:47.795806 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:47.795884 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:47.798363 (+ 2479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:47.798364 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:47.798365 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:47.798373 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:47.799823 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:47.800185 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:47.800186 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:47.800187 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:47.800252 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:47.800517 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:47.800517 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:47.800518 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:47.800617 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:47.800666 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:47.800984 (+ 318us) server_negotiation.cc:299] Negotiation successful
0928 06:37:47.801072 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 06:37:47.801314 7015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:47.792654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:45035 (local address 127.0.0.1:50480)
0928 06:37:47.792832 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:47.792863 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:47.793272 (+ 409us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:47.793463 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:47.793465 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:47.793751 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:37:47.793953 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:47.793958 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:47.794564 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:47.794564 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:47.795400 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:47.795407 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:47.795975 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:47.795977 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:47.796141 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:47.796621 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:37:47.796631 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:37:47.798282 (+ 1651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:37:47.799905 (+ 1623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:47.799910 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:47.799911 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:47.800115 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:47.800323 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:47.800324 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:47.800324 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:47.800452 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:47.800681 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:37:47.800685 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:37:47.800835 (+ 150us) client_negotiation.cc:769] Sending connection context
0928 06:37:47.800935 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 06:37:47.801082 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":104,"threads_started":1}
I20260928 06:37:47.801991 7013 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.157.125" port: 41097 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:37:47 dist-test-slave-rrj5 krb5kdc[6782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.157.124@KRBTEST.COM
Sep 28 06:37:47 dist-test-slave-rrj5 krb5kdc[6782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577467, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.124@KRBTEST.COM for kudu/127.16.157.125@KRBTEST.COM
I20260928 06:37:47.806330 7016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:47.797303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38910 (local address 127.16.157.124:44225)
0928 06:37:47.797591 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:47.797595 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:47.797620 (+ 25us) server_negotiation.cc:407] Connection header received
0928 06:37:47.797646 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:47.797648 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:47.797670 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:47.797755 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:47.798395 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:47.799331 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:47.800271 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:47.800729 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:47.800790 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:47.803394 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:47.803409 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:47.803409 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:47.803415 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:47.804984 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:47.805395 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:47.805405 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:47.805405 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:47.805470 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:47.805694 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:47.805694 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:47.805695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:47.805862 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:47.806036 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:47.806138 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 06:37:47.806224 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":74,"threads_started":1}
I20260928 06:37:47.808481 7015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:47.802239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.125:41097 (local address 127.0.0.1:47010)
0928 06:37:47.802288 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:47.802299 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:47.802339 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:47.802462 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:47.802463 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:47.802609 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:37:47.802663 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:47.802665 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:47.803343 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:47.803344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:47.803783 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:47.803784 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:47.804169 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:47.804170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:47.804236 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:47.804505 (+ 269us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:37:47.804509 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:37:47.805659 (+ 1150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:37:47.807535 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:47.807536 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:47.807536 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:47.807760 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:47.807979 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:47.807979 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:47.807980 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:47.808030 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:47.808280 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:37:47.808281 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:37:47.808320 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 06:37:47.808368 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 06:37:47.808429 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 06:37:47.808559 6942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:47.802297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47010 (local address 127.16.157.125:41097)
0928 06:37:47.802339 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:47.802341 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:47.802349 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:37:47.802406 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:47.802407 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:47.802414 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:47.802457 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:47.802741 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:47.803265 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:47.803881 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:47.804098 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:47.804170 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:47.805729 (+ 1559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:47.805730 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:47.805730 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:47.805736 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:47.807458 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:47.807844 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:47.807845 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:47.807845 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:47.807914 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:47.808092 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:47.808093 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:47.808093 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:47.808218 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:47.808281 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:47.808405 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 06:37:47.808473 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 06:37:47.808840 7013 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.157.124" port: 44225 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:37:47 dist-test-slave-rrj5 krb5kdc[6782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
Sep 28 06:37:47 dist-test-slave-rrj5 krb5kdc[6782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577467, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.124@KRBTEST.COM for kudu/127.16.157.124@KRBTEST.COM
I20260928 06:37:47.814950 6866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:47.807138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50496 (local address 127.16.157.126:45035)
0928 06:37:47.807176 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:47.807177 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:47.807184 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:37:47.807206 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:47.807207 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:47.807216 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:47.807263 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:47.807848 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:47.808528 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:47.809465 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:47.809739 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:47.809808 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:47.812074 (+ 2266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:47.812076 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:47.812076 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:47.812084 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:47.813784 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:47.814187 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:47.814188 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:47.814188 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:47.814245 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:47.814434 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:47.814435 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:47.814435 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:47.814559 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:47.814748 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:47.814785 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 06:37:47.814852 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:37:47.815939 7015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:47.809058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.124:44225 (local address 127.0.0.1:38920)
0928 06:37:47.809141 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:47.809150 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:47.809194 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:47.809292 (+ 98us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:47.809293 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:47.809395 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:37:47.809419 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:47.809420 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:47.810152 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:47.810153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:47.810973 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:47.810976 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:47.811368 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:47.811369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:47.811415 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:47.811777 (+ 362us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:37:47.811781 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:37:47.813080 (+ 1299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:37:47.814911 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:47.814912 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:47.814912 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:47.815142 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:47.815338 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:47.815339 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:47.815339 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:47.815400 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:47.815648 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:37:47.815648 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:37:47.815705 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 06:37:47.815791 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 06:37:47.815886 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260928 06:37:47.816051 7016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:47.809109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38920 (local address 127.16.157.124:44225)
0928 06:37:47.809142 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:47.809144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:47.809186 (+ 42us) server_negotiation.cc:407] Connection header received
0928 06:37:47.809234 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:47.809235 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:47.809243 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:47.809290 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:47.809497 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:47.810059 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:47.811055 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:47.811289 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:47.811360 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:47.813165 (+ 1805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:47.813166 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:47.813167 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:47.813173 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:47.814838 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:47.815212 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:47.815213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:47.815213 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:47.815270 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:47.815462 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:47.815463 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:47.815463 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:47.815572 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:47.815642 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:47.815836 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 06:37:47.815945 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":18}
I20260928 06:37:47.818261 7013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0795fc67e09240509506eceaac7fa988: Bootstrap starting.
I20260928 06:37:47.819360 7013 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0795fc67e09240509506eceaac7fa988: Neither blocks nor log segments found. Creating new log.
I20260928 06:37:47.819670 7013 log.cc:824] T 00000000000000000000000000000000 P 0795fc67e09240509506eceaac7fa988: Log is configured to *not* fsync() on all Append() calls
Sep 28 06:37:47 dist-test-slave-rrj5 krb5kdc[6782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.157.125@KRBTEST.COM
I20260928 06:37:47.820528 7013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0795fc67e09240509506eceaac7fa988: No bootstrap required, opened a new log
I20260928 06:37:47.822751 7013 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0795fc67e09240509506eceaac7fa988 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "955d82548c254fcb994c27b3628f50dd" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 45035 } } peers { permanent_uuid: "149bb1d82393435bad11024bc557224c" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 41097 } } peers { permanent_uuid: "0795fc67e09240509506eceaac7fa988" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 44225 } }
I20260928 06:37:47.822852 6942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:47.815546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47026 (local address 127.16.157.125:41097)
0928 06:37:47.815581 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:47.815583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:47.815633 (+ 50us) server_negotiation.cc:407] Connection header received
0928 06:37:47.815701 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:47.815702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:47.815710 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:47.815784 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:47.816352 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:47.816782 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:47.817650 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:47.817822 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:47.817871 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:47.820310 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:47.820311 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:47.820312 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:47.820318 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:47.821844 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:47.822138 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:47.822138 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:47.822139 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:47.822202 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:47.822369 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:47.822369 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:47.822370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:47.822505 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:47.822668 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:47.822706 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 06:37:47.822781 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:37:47.822909 7013 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0795fc67e09240509506eceaac7fa988 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0795fc67e09240509506eceaac7fa988, State: Initialized, Role: FOLLOWER
I20260928 06:37:47.822994 7013 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0795fc67e09240509506eceaac7fa988 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "955d82548c254fcb994c27b3628f50dd" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 45035 } } peers { permanent_uuid: "149bb1d82393435bad11024bc557224c" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 41097 } } peers { permanent_uuid: "0795fc67e09240509506eceaac7fa988" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 44225 } }
I20260928 06:37:47.823364 7013 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0795fc67e09240509506eceaac7fa988 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:37:47.823328 7019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0795fc67e09240509506eceaac7fa988 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "955d82548c254fcb994c27b3628f50dd" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 45035 } } peers { permanent_uuid: "149bb1d82393435bad11024bc557224c" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 41097 } } peers { permanent_uuid: "0795fc67e09240509506eceaac7fa988" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 44225 } } }
I20260928 06:37:47.823443 7019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0795fc67e09240509506eceaac7fa988 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:37:47.824507 17013 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:37:47.825129 6863 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.157.124" port: 44225 } attempt: 2
W20260928 06:37:47.825382 7030 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0795fc67e09240509506eceaac7fa988: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:37:47.825435 7030 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:37:47.825664 6837 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "955d82548c254fcb994c27b3628f50dd"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50496
I20260928 06:37:47.825632 6945 master_runner.cc:186] Error getting master registration for 127.16.157.126:45035: OK, instance_id { permanent_uuid: "955d82548c254fcb994c27b3628f50dd" instance_seqno: 1790577467479409 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 06:37:47.828506 6945 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 28 06:37:47 dist-test-slave-rrj5 krb5kdc[6782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577467, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.126@KRBTEST.COM for kudu/127.16.157.124@KRBTEST.COM
I20260928 06:37:47.832666 6865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:47.825350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.124:44225 (local address 127.0.0.1:38934)
0928 06:37:47.825440 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:47.825450 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:47.825487 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:47.825788 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:47.825789 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:47.825937 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:37:47.825978 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:47.825979 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:47.826823 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:47.826824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:47.827615 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:47.827618 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:47.828058 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:47.828059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:47.828110 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:47.828511 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:37:47.828516 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:37:47.829789 (+ 1273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:37:47.831542 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:47.831543 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:47.831543 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:47.831755 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:47.831963 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:47.831964 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:47.831965 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:47.832039 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:47.832368 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:37:47.832369 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:37:47.832411 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 06:37:47.832479 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 06:37:47.832569 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 06:37:47.832679 7016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:47.825552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38934 (local address 127.16.157.124:44225)
0928 06:37:47.825652 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:47.825654 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:47.825661 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:37:47.825680 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:47.825680 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:47.825688 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:47.825750 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:47.826055 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:47.826733 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:47.827718 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:47.827988 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:47.828058 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:47.829875 (+ 1817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:47.829876 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:47.829876 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:47.829882 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:47.831449 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:47.831830 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:47.831832 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:47.831832 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:47.831891 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:47.832124 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:47.832125 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:47.832126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:47.832285 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:47.832369 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:47.832521 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 06:37:47.832598 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 06:37:47.834425 6863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 955d82548c254fcb994c27b3628f50dd: Bootstrap starting.
I20260928 06:37:47.835121 6863 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 955d82548c254fcb994c27b3628f50dd: Neither blocks nor log segments found. Creating new log.
I20260928 06:37:47.835451 6863 log.cc:824] T 00000000000000000000000000000000 P 955d82548c254fcb994c27b3628f50dd: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:47.836334 6863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 955d82548c254fcb994c27b3628f50dd: No bootstrap required, opened a new log
I20260928 06:37:47.837702 6863 raft_consensus.cc:348] T 00000000000000000000000000000000 P 955d82548c254fcb994c27b3628f50dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "955d82548c254fcb994c27b3628f50dd" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 45035 } } peers { permanent_uuid: "149bb1d82393435bad11024bc557224c" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 41097 } } peers { permanent_uuid: "0795fc67e09240509506eceaac7fa988" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 44225 } }
I20260928 06:37:47.837837 6863 raft_consensus.cc:729] T 00000000000000000000000000000000 P 955d82548c254fcb994c27b3628f50dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 955d82548c254fcb994c27b3628f50dd, State: Initialized, Role: FOLLOWER
I20260928 06:37:47.837917 6863 consensus_queue.cc:260] T 00000000000000000000000000000000 P 955d82548c254fcb994c27b3628f50dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "955d82548c254fcb994c27b3628f50dd" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 45035 } } peers { permanent_uuid: "149bb1d82393435bad11024bc557224c" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 41097 } } peers { permanent_uuid: "0795fc67e09240509506eceaac7fa988" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 44225 } }
I20260928 06:37:47.838267 6863 sys_catalog.cc:564] T 00000000000000000000000000000000 P 955d82548c254fcb994c27b3628f50dd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:37:47.838246 7033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 955d82548c254fcb994c27b3628f50dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "955d82548c254fcb994c27b3628f50dd" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 45035 } } peers { permanent_uuid: "149bb1d82393435bad11024bc557224c" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 41097 } } peers { permanent_uuid: "0795fc67e09240509506eceaac7fa988" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 44225 } } }
I20260928 06:37:47.838296 6837 raft_consensus.cc:482] T 00000000000000000000000000000000 P 955d82548c254fcb994c27b3628f50dd [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 06:37:47.838372 6837 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 955d82548c254fcb994c27b3628f50dd [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:37:47.839098 6837 raft_consensus.cc:504] T 00000000000000000000000000000000 P 955d82548c254fcb994c27b3628f50dd [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "955d82548c254fcb994c27b3628f50dd" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 45035 } } peers { permanent_uuid: "149bb1d82393435bad11024bc557224c" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 41097 } } peers { permanent_uuid: "0795fc67e09240509506eceaac7fa988" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 44225 } }
I20260928 06:37:47.839340 6837 leader_election.cc:288] T 00000000000000000000000000000000 P 955d82548c254fcb994c27b3628f50dd [CANDIDATE]: Term 1 election: Requested vote from peers 149bb1d82393435bad11024bc557224c (127.16.157.125:41097), 0795fc67e09240509506eceaac7fa988 (127.16.157.124:44225)
I20260928 06:37:47.839437 7033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 955d82548c254fcb994c27b3628f50dd [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:37:47.839984 6987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "955d82548c254fcb994c27b3628f50dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0795fc67e09240509506eceaac7fa988"
I20260928 06:37:47.840104 6987 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0795fc67e09240509506eceaac7fa988 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:37:47.841125 6987 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0795fc67e09240509506eceaac7fa988 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 955d82548c254fcb994c27b3628f50dd in term 1.
I20260928 06:37:47.841383 6814 leader_election.cc:302] T 00000000000000000000000000000000 P 955d82548c254fcb994c27b3628f50dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0795fc67e09240509506eceaac7fa988, 955d82548c254fcb994c27b3628f50dd; no voters:
I20260928 06:37:47.841502 7033 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 955d82548c254fcb994c27b3628f50dd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:37:47.841681 7033 raft_consensus.cc:686] T 00000000000000000000000000000000 P 955d82548c254fcb994c27b3628f50dd [term 1 LEADER]: Becoming Leader. State: Replica: 955d82548c254fcb994c27b3628f50dd, State: Running, Role: LEADER
I20260928 06:37:47.841801 7033 consensus_queue.cc:237] T 00000000000000000000000000000000 P 955d82548c254fcb994c27b3628f50dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "955d82548c254fcb994c27b3628f50dd" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 45035 } } peers { permanent_uuid: "149bb1d82393435bad11024bc557224c" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 41097 } } peers { permanent_uuid: "0795fc67e09240509506eceaac7fa988" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 44225 } }
I20260928 06:37:47.842231 7033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 955d82548c254fcb994c27b3628f50dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 955d82548c254fcb994c27b3628f50dd. Latest consensus state: current_term: 1 leader_uuid: "955d82548c254fcb994c27b3628f50dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "955d82548c254fcb994c27b3628f50dd" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 45035 } } peers { permanent_uuid: "149bb1d82393435bad11024bc557224c" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 41097 } } peers { permanent_uuid: "0795fc67e09240509506eceaac7fa988" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 44225 } } }
I20260928 06:37:47.842317 7033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 955d82548c254fcb994c27b3628f50dd [sys.catalog]: This master's current role is: LEADER
I20260928 06:37:47.842563 7052 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:37:47.843096 7052 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:37:47.841876 6795 master_runner.cc:186] Error getting master registration for 127.16.157.125:41097: OK, instance_id { permanent_uuid: "149bb1d82393435bad11024bc557224c" instance_seqno: 1790577467625497 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 06:37:47.844573 6795 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 06:37:47.850494 6987 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0795fc67e09240509506eceaac7fa988 [term 1 FOLLOWER]: Refusing update from remote peer 955d82548c254fcb994c27b3628f50dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:37:47.850775 7033 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 955d82548c254fcb994c27b3628f50dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "0795fc67e09240509506eceaac7fa988" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 44225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:37:47.852033 7019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0795fc67e09240509506eceaac7fa988 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 955d82548c254fcb994c27b3628f50dd. Latest consensus state: current_term: 1 leader_uuid: "955d82548c254fcb994c27b3628f50dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "955d82548c254fcb994c27b3628f50dd" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 45035 } } peers { permanent_uuid: "149bb1d82393435bad11024bc557224c" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 41097 } } peers { permanent_uuid: "0795fc67e09240509506eceaac7fa988" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 44225 } } }
I20260928 06:37:47.852128 7019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0795fc67e09240509506eceaac7fa988 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:37:47.852615 6866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:47.840710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50500 (local address 127.16.157.126:45035)
0928 06:37:47.840746 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:47.840748 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:47.841162 (+ 414us) server_negotiation.cc:407] Connection header received
0928 06:37:47.841430 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:47.841431 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:47.841441 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:47.841473 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:47.843381 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:47.844069 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:47.846122 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:47.846393 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:47.846446 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:47.848874 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:47.848876 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:47.848876 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:47.848883 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:47.850560 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:47.851276 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:47.851277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:47.851277 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:47.851346 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:47.851664 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:47.851665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:47.851665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:47.851798 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:47.851881 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:47.852447 (+ 566us) server_negotiation.cc:299] Negotiation successful
0928 06:37:47.852527 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:37:47.854838 7016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:47.845139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38950 (local address 127.16.157.124:44225)
0928 06:37:47.845441 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:47.845443 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:47.845685 (+ 242us) server_negotiation.cc:407] Connection header received
0928 06:37:47.845770 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:47.845771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:47.845780 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:47.845808 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:47.846498 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:47.847178 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:47.848440 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:47.848731 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:47.848780 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:47.850681 (+ 1901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:47.850682 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:47.850682 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:47.850689 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:47.852342 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:47.853737 (+ 1395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:47.853738 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:47.853738 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:47.853805 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:47.854124 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:47.854125 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:47.854126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:47.854256 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:47.854353 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:47.854677 (+ 324us) server_negotiation.cc:299] Negotiation successful
0928 06:37:47.854755 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284}
I20260928 06:37:47.856269 7033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 955d82548c254fcb994c27b3628f50dd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "955d82548c254fcb994c27b3628f50dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "955d82548c254fcb994c27b3628f50dd" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 45035 } } peers { permanent_uuid: "149bb1d82393435bad11024bc557224c" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 41097 } } peers { permanent_uuid: "0795fc67e09240509506eceaac7fa988" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 44225 } } }
I20260928 06:37:47.856359 7033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 955d82548c254fcb994c27b3628f50dd [sys.catalog]: This master's current role is: LEADER
I20260928 06:37:47.856653 7019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0795fc67e09240509506eceaac7fa988 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "955d82548c254fcb994c27b3628f50dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "955d82548c254fcb994c27b3628f50dd" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 45035 } } peers { permanent_uuid: "149bb1d82393435bad11024bc557224c" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 41097 } } peers { permanent_uuid: "0795fc67e09240509506eceaac7fa988" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 44225 } } }
I20260928 06:37:47.856732 7019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0795fc67e09240509506eceaac7fa988 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:37:47.857069 7057 mvcc.cc:205] Tried to move back new op lower bound from 7334205308313018368 to 7334205308280786944. Current Snapshot: MvccSnapshot[applied={T|T < 7334205308313018368}]
I20260928 06:37:47.858611 6938 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.157.124" port: 44225 } attempt: 3
I20260928 06:37:47.862044 7052 catalog_manager.cc:1347] Generated new cluster ID: 95ad4621813d42c991773bb288519ff9
I20260928 06:37:47.862092 7052 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:37:47.862720 6942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:47.848342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47028 (local address 127.16.157.125:41097)
0928 06:37:47.848390 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:47.848392 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:47.853447 (+ 5055us) server_negotiation.cc:407] Connection header received
0928 06:37:47.853597 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:47.853598 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:47.853621 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:47.853652 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:47.854544 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:47.855287 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:47.856954 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:47.857230 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:47.857301 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:47.858691 (+ 1390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:47.858692 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:47.858693 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:47.858700 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:47.860530 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:47.860938 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:47.860939 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:47.860939 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:47.860998 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:47.861228 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:47.861229 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:47.861230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:47.861368 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:47.861455 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:47.861678 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 06:37:47.862620 (+ 942us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
Sep 28 06:37:47 dist-test-slave-rrj5 krb5kdc[6782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577467, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.125@KRBTEST.COM for kudu/127.16.157.124@KRBTEST.COM
I20260928 06:37:47.873420 7052 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:37:47.873780 7016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:47.858934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38964 (local address 127.16.157.124:44225)
0928 06:37:47.858975 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:47.858977 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:47.858984 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:37:47.859003 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:47.859004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:47.859013 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:47.859082 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:47.859421 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:47.860190 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:47.861142 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:47.861435 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:47.861495 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:47.870649 (+ 9154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:47.870650 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:47.870651 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:47.870657 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:47.872218 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:47.872641 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:47.872642 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:47.872642 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:47.872708 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:47.872938 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:47.872939 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:47.872940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:47.873087 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:47.873141 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:47.873610 (+ 469us) server_negotiation.cc:299] Negotiation successful
0928 06:37:47.873700 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 06:37:47.874015 6940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:47.858817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.124:44225 (local address 127.0.0.1:38964)
0928 06:37:47.858899 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:47.858908 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:47.858942 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:47.859099 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:47.859099 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:47.859275 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:37:47.859339 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:47.859341 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:47.860270 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:47.860270 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:47.861067 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:47.861069 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:47.861545 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:47.861546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:47.861639 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:47.862064 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:37:47.862069 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:37:47.870557 (+ 8488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:37:47.872336 (+ 1779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:47.872337 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:47.872337 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:47.872552 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:47.872788 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:47.872789 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:47.872790 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:47.872859 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:47.873184 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:37:47.873185 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:37:47.873251 (+ 66us) client_negotiation.cc:769] Sending connection context
0928 06:37:47.873319 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 06:37:47.873407 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260928 06:37:47.874392 7052 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:37:47.875463 6938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 149bb1d82393435bad11024bc557224c: Bootstrap starting.
I20260928 06:37:47.876122 6938 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 149bb1d82393435bad11024bc557224c: Neither blocks nor log segments found. Creating new log.
I20260928 06:37:47.876401 6938 log.cc:824] T 00000000000000000000000000000000 P 149bb1d82393435bad11024bc557224c: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:47.877529 6938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 149bb1d82393435bad11024bc557224c: No bootstrap required, opened a new log
I20260928 06:37:47.881026 6938 raft_consensus.cc:348] T 00000000000000000000000000000000 P 149bb1d82393435bad11024bc557224c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "955d82548c254fcb994c27b3628f50dd" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 45035 } } peers { permanent_uuid: "149bb1d82393435bad11024bc557224c" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 41097 } } peers { permanent_uuid: "0795fc67e09240509506eceaac7fa988" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 44225 } }
I20260928 06:37:47.881284 6938 raft_consensus.cc:729] T 00000000000000000000000000000000 P 149bb1d82393435bad11024bc557224c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149bb1d82393435bad11024bc557224c, State: Initialized, Role: FOLLOWER
I20260928 06:37:47.881479 6938 consensus_queue.cc:260] T 00000000000000000000000000000000 P 149bb1d82393435bad11024bc557224c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "955d82548c254fcb994c27b3628f50dd" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 45035 } } peers { permanent_uuid: "149bb1d82393435bad11024bc557224c" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 41097 } } peers { permanent_uuid: "0795fc67e09240509506eceaac7fa988" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 44225 } }
I20260928 06:37:47.882100 6911 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 149bb1d82393435bad11024bc557224c [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:37:47.882974 6911 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 149bb1d82393435bad11024bc557224c [term 1 FOLLOWER]: Refusing update from remote peer 955d82548c254fcb994c27b3628f50dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:37:47.883253 7050 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 955d82548c254fcb994c27b3628f50dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "149bb1d82393435bad11024bc557224c" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 41097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:37:47.882121 6912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "955d82548c254fcb994c27b3628f50dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "149bb1d82393435bad11024bc557224c"
I20260928 06:37:47.884274 6912 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 149bb1d82393435bad11024bc557224c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 955d82548c254fcb994c27b3628f50dd in term 1.
I20260928 06:37:47.882027 6938 sys_catalog.cc:564] T 00000000000000000000000000000000 P 149bb1d82393435bad11024bc557224c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:37:47.889742 7064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 149bb1d82393435bad11024bc557224c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 955d82548c254fcb994c27b3628f50dd. Latest consensus state: current_term: 1 leader_uuid: "955d82548c254fcb994c27b3628f50dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "955d82548c254fcb994c27b3628f50dd" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 45035 } } peers { permanent_uuid: "149bb1d82393435bad11024bc557224c" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 41097 } } peers { permanent_uuid: "0795fc67e09240509506eceaac7fa988" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 44225 } } }
I20260928 06:37:47.889870 7064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 149bb1d82393435bad11024bc557224c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:37:47.889953 7064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 149bb1d82393435bad11024bc557224c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "955d82548c254fcb994c27b3628f50dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "955d82548c254fcb994c27b3628f50dd" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 45035 } } peers { permanent_uuid: "149bb1d82393435bad11024bc557224c" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 41097 } } peers { permanent_uuid: "0795fc67e09240509506eceaac7fa988" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 44225 } } }
I20260928 06:37:47.890021 7064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 149bb1d82393435bad11024bc557224c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:37:47.890573 7065 mvcc.cc:205] Tried to move back new op lower bound from 7334205308404019200 to 7334205308280786944. Current Snapshot: MvccSnapshot[applied={T|T < 7334205308313018368}]
W20260928 06:37:47.891769 7078 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 149bb1d82393435bad11024bc557224c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:37:47.891872 7078 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:37:47.889849 7063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 149bb1d82393435bad11024bc557224c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "955d82548c254fcb994c27b3628f50dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "955d82548c254fcb994c27b3628f50dd" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 45035 } } peers { permanent_uuid: "149bb1d82393435bad11024bc557224c" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 41097 } } peers { permanent_uuid: "0795fc67e09240509506eceaac7fa988" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 44225 } } }
I20260928 06:37:47.893695 7063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 149bb1d82393435bad11024bc557224c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:37:47.898495 7050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 955d82548c254fcb994c27b3628f50dd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "955d82548c254fcb994c27b3628f50dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "955d82548c254fcb994c27b3628f50dd" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 45035 } } peers { permanent_uuid: "149bb1d82393435bad11024bc557224c" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 41097 } } peers { permanent_uuid: "0795fc67e09240509506eceaac7fa988" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 44225 } } }
I20260928 06:37:47.898593 7050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 955d82548c254fcb994c27b3628f50dd [sys.catalog]: This master's current role is: LEADER
I20260928 06:37:47.900132 7052 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 955d82548c254fcb994c27b3628f50dd: Generated new TSK 0
I20260928 06:37:47.900343 7052 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 06:37:47.928994 7016 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:37:47.927594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38970 (local address 127.16.157.124:44225)
0928 06:37:47.927642 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:47.927644 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:47.928281 (+ 637us) server_negotiation.cc:407] Connection header received
0928 06:37:47.928388 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:47.928389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:47.928400 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:47.928431 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:47.928927 (+ 496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38970: BlockingRecv error: recv got EOF from 127.0.0.1:38970 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 06:37:47.929745 6866 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:37:47.927990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50514 (local address 127.16.157.126:45035)
0928 06:37:47.928029 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:47.928032 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:47.929221 (+ 1189us) server_negotiation.cc:407] Connection header received
0928 06:37:47.929313 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:47.929314 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:47.929325 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:47.929353 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:47.929682 (+ 329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50514: BlockingRecv error: recv got EOF from 127.0.0.1:50514 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 06:37:47.930779 6942 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:37:47.930092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47038 (local address 127.16.157.125:41097)
0928 06:37:47.930134 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:47.930136 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:47.930257 (+ 121us) server_negotiation.cc:407] Connection header received
0928 06:37:47.930360 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:47.930361 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:47.930371 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:47.930399 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:47.930721 (+ 322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47038: BlockingRecv error: recv got EOF from 127.0.0.1:47038 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 06:37:47.934617 6942 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:37:47.933856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47042 (local address 127.16.157.125:41097)
0928 06:37:47.933892 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:47.933894 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:47.934074 (+ 180us) server_negotiation.cc:407] Connection header received
0928 06:37:47.934183 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:47.934184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:47.934204 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:47.934233 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:47.934568 (+ 335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47042: BlockingRecv error: recv got EOF from 127.0.0.1:47042 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 06:37:47.937731 6942 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:37:47.936933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47056 (local address 127.16.157.125:41097)
0928 06:37:47.936967 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:47.936969 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:47.937177 (+ 208us) server_negotiation.cc:407] Connection header received
0928 06:37:47.937283 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:47.937283 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:47.937293 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:47.937322 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:47.937684 (+ 362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47056: BlockingRecv error: recv got EOF from 127.0.0.1:47056 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 06:37:47.942591 6942 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:37:47.941804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47060 (local address 127.16.157.125:41097)
0928 06:37:47.941848 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:47.941850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:47.942105 (+ 255us) server_negotiation.cc:407] Connection header received
0928 06:37:47.942200 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:47.942201 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:47.942210 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:47.942240 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:47.942544 (+ 304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47060: BlockingRecv error: recv got EOF from 127.0.0.1:47060 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260928 06:37:47.948444 6942 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:37:47.947692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47064 (local address 127.16.157.125:41097)
0928 06:37:47.947726 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:47.947728 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:47.947968 (+ 240us) server_negotiation.cc:407] Connection header received
0928 06:37:47.948060 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:47.948061 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:47.948071 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:47.948100 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:47.948402 (+ 302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47064: BlockingRecv error: recv got EOF from 127.0.0.1:47064 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 06:37:47.958391 6942 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:37:47.957623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47074 (local address 127.16.157.125:41097)
0928 06:37:47.957661 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:47.957663 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:47.957885 (+ 222us) server_negotiation.cc:407] Connection header received
0928 06:37:47.957980 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:47.957982 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:47.957992 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:47.958023 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:47.958342 (+ 319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47074: BlockingRecv error: recv got EOF from 127.0.0.1:47074 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 06:37:47.976518 6942 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:37:47.975663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47086 (local address 127.16.157.125:41097)
0928 06:37:47.975708 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:47.975710 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:47.975961 (+ 251us) server_negotiation.cc:407] Connection header received
0928 06:37:47.976063 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:47.976064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:47.976075 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:47.976104 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:47.976465 (+ 361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47086: BlockingRecv error: recv got EOF from 127.0.0.1:47086 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 06:37:48.011962 6942 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:37:48.009981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47090 (local address 127.16.157.125:41097)
0928 06:37:48.010035 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:48.010037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:48.010305 (+ 268us) server_negotiation.cc:407] Connection header received
0928 06:37:48.011407 (+ 1102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:48.011409 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:48.011425 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:48.011467 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:48.011885 (+ 418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47090: BlockingRecv error: recv got EOF from 127.0.0.1:47090 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260928 06:37:48.078253 6942 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:37:48.077493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47092 (local address 127.16.157.125:41097)
0928 06:37:48.077627 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:48.077629 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:48.077661 (+ 32us) server_negotiation.cc:407] Connection header received
0928 06:37:48.077741 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:48.077742 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:48.077753 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:48.077807 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:48.078163 (+ 356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47092: BlockingRecv error: recv got EOF from 127.0.0.1:47092 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
W20260928 06:37:48.208206 6942 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:37:48.207444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47102 (local address 127.16.157.125:41097)
0928 06:37:48.207539 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:48.207541 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:48.207635 (+ 94us) server_negotiation.cc:407] Connection header received
0928 06:37:48.207713 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:48.207714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:48.207725 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:48.207800 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:48.208141 (+ 341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47102: BlockingRecv error: recv got EOF from 127.0.0.1:47102 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260928 06:37:48.466065 6942 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:37:48.465411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43100 (local address 127.16.157.125:41097)
0928 06:37:48.465487 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:48.465489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:48.465503 (+ 14us) server_negotiation.cc:407] Connection header received
0928 06:37:48.465588 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:48.465589 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:48.465599 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:48.465651 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:48.465996 (+ 345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43100: BlockingRecv error: recv got EOF from 127.0.0.1:43100 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 06:37:48.825884 7030 catalog_manager.cc:1259] Loaded cluster ID: 95ad4621813d42c991773bb288519ff9
I20260928 06:37:48.825985 7030 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0795fc67e09240509506eceaac7fa988: loading cluster ID for follower catalog manager: success
I20260928 06:37:48.826812 7030 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0795fc67e09240509506eceaac7fa988: acquiring CA information for follower catalog manager: success
I20260928 06:37:48.827118 7030 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0795fc67e09240509506eceaac7fa988: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 06:37:48.892375 7078 catalog_manager.cc:1259] Loaded cluster ID: 95ad4621813d42c991773bb288519ff9
I20260928 06:37:48.892453 7078 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 149bb1d82393435bad11024bc557224c: loading cluster ID for follower catalog manager: success
I20260928 06:37:48.893239 7078 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 149bb1d82393435bad11024bc557224c: acquiring CA information for follower catalog manager: success
I20260928 06:37:48.893508 7078 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 149bb1d82393435bad11024bc557224c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 06:37:48.981935 7147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:48.979213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43116 (local address 127.16.157.125:41097)
0928 06:37:48.979328 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:48.979330 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:48.979342 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:37:48.979424 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:48.979425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:48.979492 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:48.979582 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:37:48.979737 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:48.980264 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:48.981085 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:48.981303 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:48.981393 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:48.981585 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:37:48.981697 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:37:48.981790 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:48.981812 (+ 22us) server_negotiation.cc:299] Negotiation successful
0928 06:37:48.981836 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":53,"threads_started":1}
I20260928 06:37:48.986177 7153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:48.983757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36498 (local address 127.16.157.124:44225)
0928 06:37:48.983871 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:48.983873 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:48.983883 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:37:48.983920 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:48.983920 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:48.983956 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:48.984022 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:37:48.984123 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:48.984593 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:48.985313 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:48.985496 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:48.985544 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:48.985752 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:37:48.985935 (+ 183us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:37:48.986027 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:48.986068 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 06:37:48.986090 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":57,"threads_started":1}
I20260928 06:37:48.990454 7147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:48.987726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43120 (local address 127.16.157.125:41097)
0928 06:37:48.987861 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:48.987862 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:48.987871 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:37:48.987906 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:48.987907 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:48.987916 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:48.987964 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:37:48.988106 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:48.988783 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:48.989720 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:48.989986 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:48.990042 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:48.990207 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:37:48.990239 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:37:48.990292 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:48.990359 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 06:37:48.990379 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260928 06:37:48.990772 7153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:48.987944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36504 (local address 127.16.157.124:44225)
0928 06:37:48.987997 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:48.987998 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:48.988117 (+ 119us) server_negotiation.cc:407] Connection header received
0928 06:37:48.988162 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:48.988163 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:48.988171 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:48.988235 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:37:48.988337 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:48.988784 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:48.989870 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:48.990146 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:48.990202 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:48.990394 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:37:48.990422 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:37:48.990450 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:48.990566 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 06:37:48.990585 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 06:37:48.991313 7160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:48.987774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51924 (local address 127.16.157.126:45035)
0928 06:37:48.988054 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:48.988056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:48.988063 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:37:48.988110 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:48.988111 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:48.988162 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:48.988235 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:37:48.988417 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:48.989118 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:48.990037 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:48.990314 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:48.990372 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:48.990979 (+ 607us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:37:48.991074 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:37:48.991144 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:48.991210 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 06:37:48.991231 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":49,"threads_started":1}
I20260928 06:37:48.992910 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 6795
I20260928 06:37:48.998430 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 6870
I20260928 06:37:49.003482 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 6945
2026-09-28T06:37:49Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2797 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:37:49 dist-test-slave-rrj5 krb5kdc[7164](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:37:49 dist-test-slave-rrj5 krb5kdc[7164](info): set up 2 sockets
Sep 28 06:37:49 dist-test-slave-rrj5 krb5kdc[7164](info): commencing operation
krb5kdc: starting...
W20260928 06:37:50.022699 17013 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/krb5kdc/test-user.keytab.
Sep 28 06:37:50 dist-test-slave-rrj5 krb5kdc[7164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:37:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:37:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.157.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab.
Entry for principal kudu/127.16.157.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab.
I20260928 06:37:50.140946 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:40389
--webserver_interface=127.16.157.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=kudu/127.16.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:43701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.157.126:40389,127.16.157.125:41087,127.16.157.124:45855
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:37:50.223591 7177 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 06:37:50.223812 7177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:50.223830 7177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:50.223852 7177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:50.224972 7177 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:37:50.225010 7177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:50.225025 7177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:50.225037 7177 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:37:50.225050 7177 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:37:50.226599 7177 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:43701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.157.126:40389,127.16.157.125:41087,127.16.157.124:45855
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.126.keytab
--principal=kudu/127.16.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.126:40389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:37:50.226853 7177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:50.227111 7177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:50.232209 7182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:50.232156 7183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:50.232182 7185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:50.232492 7177 server_base.cc:1034] running on GCE node
I20260928 06:37:50.232708 7177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:50.232962 7177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:50.234102 7177 hybrid_clock.cc:648] HybridClock initialized: now 1790577470234082 us; error 33 us; skew 500 ppm
Sep 28 06:37:50 dist-test-slave-rrj5 krb5kdc[7164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577470, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:50.236565 7177 init.cc:377] Logged in from keytab as kudu/127.16.157.126@KRBTEST.COM (short username kudu)
I20260928 06:37:50.237002 7177 webserver.cc:460] Webserver started at http://127.16.157.126:34157/ using document root <none> and password file <none>
I20260928 06:37:50.237249 7177 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:50.237318 7177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:50.237471 7177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:37:50.238770 7177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/minicluster-data/master-0/data/instance:
uuid: "e15fdbb090ae45729cd66cb9c7bf0ccf"
format_stamp: "Formatted at 2026-09-28 06:37:50 on dist-test-slave-rrj5"
server_key: "b489a1e02b0ecf7211214ac0c20193c0"
server_key_iv: "a96dbfce66dbc88c63b0b93a3ca48f85"
server_key_version: "encryptionkey@0"
I20260928 06:37:50.239147 7177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/minicluster-data/master-0/wal/instance:
uuid: "e15fdbb090ae45729cd66cb9c7bf0ccf"
format_stamp: "Formatted at 2026-09-28 06:37:50 on dist-test-slave-rrj5"
server_key: "b489a1e02b0ecf7211214ac0c20193c0"
server_key_iv: "a96dbfce66dbc88c63b0b93a3ca48f85"
server_key_version: "encryptionkey@0"
I20260928 06:37:50.240473 7177 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:37:50.241183 7192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:50.241402 7177 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:37:50.241496 7177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/minicluster-data/master-0/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/minicluster-data/master-0/wal
uuid: "e15fdbb090ae45729cd66cb9c7bf0ccf"
format_stamp: "Formatted at 2026-09-28 06:37:50 on dist-test-slave-rrj5"
server_key: "b489a1e02b0ecf7211214ac0c20193c0"
server_key_iv: "a96dbfce66dbc88c63b0b93a3ca48f85"
server_key_version: "encryptionkey@0"
I20260928 06:37:50.241550 7177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:50.251605 7177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:50.253399 7177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:50.253510 7177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:50.257607 7177 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.126:40389
I20260928 06:37:50.257673 7244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.126:40389 every 8 connection(s)
I20260928 06:37:50.258023 7177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/minicluster-data/master-0/data/info.pb
I20260928 06:37:50.258940 7245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:37:50.260524 7245 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40389 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:37:50.265767 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 7177
I20260928 06:37:50.265846 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/minicluster-data/master-0/wal/instance
I20260928 06:37:50.266062 17013 external_mini_cluster.cc:1428] Setting key 9ea38bca0124e5583b0b60eae82bb9ea
Sep 28 06:37:50 dist-test-slave-rrj5 krb5kdc[7164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577470, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.126@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:37:50.271231 7247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:50.261257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40389 (local address 127.0.0.1:60220)
0928 06:37:50.261445 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:50.261557 (+ 112us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:50.262111 (+ 554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:50.262418 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:50.262421 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:50.262673 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:37:50.262831 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:50.262835 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:50.263854 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:50.263855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:50.264541 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:50.264543 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:50.265025 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:50.265025 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:50.265087 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:50.265696 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:37:50.265705 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:37:50.267494 (+ 1789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:37:50.269639 (+ 2145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:50.269660 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:50.269661 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:50.269906 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:50.270196 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:50.270197 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:50.270197 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:50.270299 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:50.270777 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:37:50.270783 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:37:50.270844 (+ 61us) client_negotiation.cc:769] Sending connection context
0928 06:37:50.270923 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 06:37:50.271055 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":90,"threads_started":1}
I20260928 06:37:50.271332 7248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:50.261307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60220 (local address 127.16.157.126:40389)
0928 06:37:50.261536 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:50.261551 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:50.261730 (+ 179us) server_negotiation.cc:407] Connection header received
0928 06:37:50.262293 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:50.262294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:50.262311 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:50.262388 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:50.262910 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:50.263753 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:50.264620 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:50.264944 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:50.265025 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:50.267602 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:50.267611 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:50.267617 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:50.267625 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:50.269519 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:50.269999 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:50.270002 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:50.270003 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:50.270113 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:50.270370 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:50.270371 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:50.270371 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:50.270690 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:50.270793 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:50.270990 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 06:37:50.271149 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":201,"thread_start_us":132,"threads_started":1}
I20260928 06:37:50.272099 7245 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.157.125" port: 41087 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:37:50.272487 7247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:50.272369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.125:41087 (local address 127.0.0.1:33240)
0928 06:37:50.272420 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:50.272469 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.125:41087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 06:37:50.272619 7194 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.157.125:41087: connect: Connection refused (error 111)
I20260928 06:37:50.273077 7247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:50.272984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.125:41087 (local address 127.0.0.1:33250)
0928 06:37:50.273029 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:50.273059 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.125:41087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 06:37:50.273183 7245 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.157.125:41087: Network error: Client connection negotiation failed: client connection to 127.16.157.125:41087: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.16.157.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.125.keytab.
Entry for principal kudu/127.16.157.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.125.keytab.
I20260928 06:37:50.292001 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.125:41087
--webserver_interface=127.16.157.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.125.keytab
--principal=kudu/127.16.157.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:43701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.157.126:40389,127.16.157.125:41087,127.16.157.124:45855
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:37:50.322400 7245 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.157.125" port: 41087 } attempt: 1
I20260928 06:37:50.322914 7247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:50.322768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.125:41087 (local address 127.0.0.1:33254)
0928 06:37:50.322835 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:50.322883 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.125:41087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 06:37:50.323267 7247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:50.323172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.125:41087 (local address 127.0.0.1:33262)
0928 06:37:50.323218 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:50.323241 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.125:41087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 06:37:50.323354 7245 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.157.125:41087: Network error: Client connection negotiation failed: client connection to 127.16.157.125:41087: connect: Connection refused (error 111)
W20260928 06:37:50.373071 7252 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 06:37:50.373281 7252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:50.373297 7252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:50.373317 7252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:50.374509 7252 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:37:50.374548 7252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:50.374562 7252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:50.374614 7252 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:37:50.374630 7252 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:37:50.376116 7252 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:43701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.157.126:40389,127.16.157.125:41087,127.16.157.124:45855
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.125.keytab
--principal=kudu/127.16.157.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.125:41087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.157.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:37:50.376361 7252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:50.376597 7252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:50.381479 7258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:50.381544 7252 server_base.cc:1034] running on GCE node
W20260928 06:37:50.381490 7257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:50.381512 7260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:50.381899 7252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:50.382193 7252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:50.383363 7252 hybrid_clock.cc:648] HybridClock initialized: now 1790577470383347 us; error 27 us; skew 500 ppm
Sep 28 06:37:50 dist-test-slave-rrj5 krb5kdc[7164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577470, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:50.385783 7252 init.cc:377] Logged in from keytab as kudu/127.16.157.125@KRBTEST.COM (short username kudu)
I20260928 06:37:50.386252 7252 webserver.cc:460] Webserver started at http://127.16.157.125:39903/ using document root <none> and password file <none>
I20260928 06:37:50.386473 7252 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:50.386538 7252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:50.386686 7252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:37:50.387744 7252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/minicluster-data/master-1/data/instance:
uuid: "84431694103a4a939c972cbca69358d7"
format_stamp: "Formatted at 2026-09-28 06:37:50 on dist-test-slave-rrj5"
server_key: "4b3f714aad5a1815edad387e57d6e8cc"
server_key_iv: "d67d912ff28f6d90f87da6532b0e3536"
server_key_version: "encryptionkey@0"
I20260928 06:37:50.388098 7252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/minicluster-data/master-1/wal/instance:
uuid: "84431694103a4a939c972cbca69358d7"
format_stamp: "Formatted at 2026-09-28 06:37:50 on dist-test-slave-rrj5"
server_key: "4b3f714aad5a1815edad387e57d6e8cc"
server_key_iv: "d67d912ff28f6d90f87da6532b0e3536"
server_key_version: "encryptionkey@0"
I20260928 06:37:50.389402 7252 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:37:50.390264 7267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:50.390460 7252 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:37:50.390566 7252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/minicluster-data/master-1/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/minicluster-data/master-1/wal
uuid: "84431694103a4a939c972cbca69358d7"
format_stamp: "Formatted at 2026-09-28 06:37:50 on dist-test-slave-rrj5"
server_key: "4b3f714aad5a1815edad387e57d6e8cc"
server_key_iv: "d67d912ff28f6d90f87da6532b0e3536"
server_key_version: "encryptionkey@0"
I20260928 06:37:50.390650 7252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:50.391481 7245 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.157.125" port: 41087 } attempt: 2
I20260928 06:37:50.391846 7247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:50.391723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.125:41087 (local address 127.0.0.1:33270)
0928 06:37:50.391782 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:50.391819 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.125:41087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 06:37:50.392171 7247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:50.392078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.125:41087 (local address 127.0.0.1:33272)
0928 06:37:50.392121 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:50.392145 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.125:41087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 06:37:50.392256 7245 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.157.125:41087: Network error: Client connection negotiation failed: client connection to 127.16.157.125:41087: connect: Connection refused (error 111)
I20260928 06:37:50.408828 7252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:50.410763 7252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:50.410935 7252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:50.415050 7252 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.125:41087
I20260928 06:37:50.415098 7319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.125:41087 every 8 connection(s)
I20260928 06:37:50.415463 7252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/minicluster-data/master-1/data/info.pb
I20260928 06:37:50.416299 7320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:37:50.417214 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 7252
I20260928 06:37:50.417348 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/minicluster-data/master-1/wal/instance
I20260928 06:37:50.417523 17013 external_mini_cluster.cc:1428] Setting key 61155b608770323fc78712547dfcc2e6
I20260928 06:37:50.418006 7320 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40389 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:37:50 dist-test-slave-rrj5 krb5kdc[7164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577470, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.125@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.157.124@KRBTEST.COM; defaulting to no policy
I20260928 06:37:50.427812 7248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:50.418730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60234 (local address 127.16.157.126:40389)
0928 06:37:50.418795 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:50.418797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:50.419015 (+ 218us) server_negotiation.cc:407] Connection header received
0928 06:37:50.419507 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:50.419509 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:50.419526 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:50.419610 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:50.420193 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:50.420975 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:50.422074 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:50.422448 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:50.422536 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:50.424960 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:50.424961 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:50.424961 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:50.424968 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:50.426411 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:50.426836 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:50.426838 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:50.426838 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:50.426905 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:50.427164 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:50.427165 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:50.427165 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:50.427284 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:50.427354 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:50.427628 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 06:37:50.427712 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 06:37:50.427976 7323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:50.418713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40389 (local address 127.0.0.1:60234)
0928 06:37:50.418884 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:50.418926 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:50.419403 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:50.419660 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:50.419663 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:50.419906 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:37:50.420092 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:50.420096 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:50.421095 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:50.421097 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:50.421983 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:50.421986 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:50.422546 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:50.422547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:50.422647 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:50.423192 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:37:50.423201 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:37:50.424851 (+ 1650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:37:50.426489 (+ 1638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:50.426496 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:50.426497 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:50.426748 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:50.426978 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:50.426979 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:50.426980 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:50.427085 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:50.427358 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:37:50.427366 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:37:50.427495 (+ 129us) client_negotiation.cc:769] Sending connection context
0928 06:37:50.427596 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 06:37:50.427749 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":78,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.157.124@KRBTEST.COM" created.
I20260928 06:37:50.428664 7320 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.157.125" port: 41087 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:37:50 dist-test-slave-rrj5 krb5kdc[7164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577470, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.125@KRBTEST.COM for kudu/127.16.157.125@KRBTEST.COM
I20260928 06:37:50.437036 7323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:50.428960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.125:41087 (local address 127.0.0.1:33280)
0928 06:37:50.429007 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:50.429017 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:50.429053 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:50.430002 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:50.430004 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:50.430169 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:37:50.430232 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:50.430234 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:50.431191 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:50.431191 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:50.431676 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:50.431677 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:50.432358 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:50.432359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:50.432413 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:50.432814 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:37:50.432819 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:37:50.434008 (+ 1189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:37:50.436024 (+ 2016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:50.436025 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:50.436025 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:50.436184 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:50.436439 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:50.436439 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:50.436440 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:50.436490 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:50.436816 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:37:50.436817 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:37:50.436850 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 06:37:50.436896 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 06:37:50.436973 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 06:37:50.437141 7324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:50.429663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33280 (local address 127.16.157.125:41087)
0928 06:37:50.429814 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:50.429820 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:50.429850 (+ 30us) server_negotiation.cc:407] Connection header received
0928 06:37:50.429875 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:50.429877 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:50.429893 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:50.429984 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:50.430309 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:50.431101 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:50.431761 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:50.432282 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:50.432362 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:50.434093 (+ 1731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:50.434102 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:50.434102 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:50.434110 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:50.435928 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:50.436280 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:50.436284 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:50.436285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:50.436355 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:50.436554 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:50.436555 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:50.436555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:50.436742 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:50.436826 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:50.436929 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 06:37:50.437023 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":112,"thread_start_us":72,"threads_started":1}
I20260928 06:37:50.437772 7320 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.157.124" port: 45855 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:37:50.438138 7323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:50.438037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.124:45855 (local address 127.0.0.1:45634)
0928 06:37:50.438080 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:50.438120 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.124:45855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 06:37:50.438305 7270 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.157.124:45855: connect: Connection refused (error 111)
I20260928 06:37:50.438845 7323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:50.438761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.124:45855 (local address 127.0.0.1:45640)
0928 06:37:50.438793 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:50.438823 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.124:45855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260928 06:37:50.438982 7320 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.157.124:45855: Network error: Client connection negotiation failed: client connection to 127.16.157.124:45855: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.157.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.124.keytab.
Entry for principal kudu/127.16.157.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.124.keytab.
I20260928 06:37:50.445344 17013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGDaHLl/build/release/bin/kudu
/tmp/dist-test-taskGDaHLl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.124:45855
--webserver_interface=127.16.157.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.124.keytab
--principal=kudu/127.16.157.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.157.84:43701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.157.126:40389,127.16.157.125:41087,127.16.157.124:45855
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:37:50.483412 7245 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.157.125" port: 41087 } attempt: 3
I20260928 06:37:50.488209 7320 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.157.124" port: 45855 } attempt: 1
Sep 28 06:37:50 dist-test-slave-rrj5 krb5kdc[7164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577470, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.126@KRBTEST.COM for kudu/127.16.157.125@KRBTEST.COM
I20260928 06:37:50.488591 7323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:50.488474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.124:45855 (local address 127.0.0.1:45652)
0928 06:37:50.488528 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:50.488564 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.124:45855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 06:37:50.488962 7323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:50.488859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.124:45855 (local address 127.0.0.1:45668)
0928 06:37:50.488912 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:50.488939 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.124:45855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 06:37:50.489092 7320 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.157.124:45855: Network error: Client connection negotiation failed: client connection to 127.16.157.124:45855: connect: Connection refused (error 111)
I20260928 06:37:50.491677 7247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:50.484093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.125:41087 (local address 127.0.0.1:33288)
0928 06:37:50.484144 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:50.484165 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:50.484263 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:50.484434 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:50.484435 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:50.484611 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:37:50.484678 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:50.484679 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:50.485639 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:50.485641 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:50.486203 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:50.486206 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:50.486654 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:50.486655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:50.486754 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:50.487246 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:37:50.487250 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:37:50.488666 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:37:50.490528 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:50.490529 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:50.490530 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:50.490826 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:50.491066 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:50.491066 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:50.491067 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:50.491166 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:50.491430 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:37:50.491431 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:37:50.491473 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 06:37:50.491525 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 06:37:50.491603 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 06:37:50.491736 7324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:50.484178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33288 (local address 127.16.157.125:41087)
0928 06:37:50.484258 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:50.484261 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:50.484273 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:37:50.484338 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:50.484339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:50.484353 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:50.484427 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:50.484763 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:50.485312 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:50.486335 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:50.486576 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:50.486656 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:50.488781 (+ 2125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:50.488782 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:50.488782 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:50.488790 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:50.490433 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:50.490922 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:50.490924 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:50.490924 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:50.491006 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:50.491231 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:50.491232 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:50.491232 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:50.491367 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:50.491436 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:50.491562 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 06:37:50.491650 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 06:37:50.492246 7245 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.157.124" port: 45855 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:37:50.492575 7247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:50.492472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.124:45855 (local address 127.0.0.1:45672)
0928 06:37:50.492523 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:50.492553 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.124:45855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 06:37:50.492853 7247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:50.492772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.124:45855 (local address 127.0.0.1:45682)
0928 06:37:50.492814 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:50.492838 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.124:45855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 06:37:50.492928 7245 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.157.124:45855: Network error: Client connection negotiation failed: client connection to 127.16.157.124:45855: connect: Connection refused (error 111)
I20260928 06:37:50.524053 7245 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.157.124" port: 45855 } attempt: 1
I20260928 06:37:50.524554 7247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:50.524392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.124:45855 (local address 127.0.0.1:45692)
0928 06:37:50.524469 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:50.524516 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.124:45855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 06:37:50.524956 7247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:50.524843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.124:45855 (local address 127.0.0.1:45702)
0928 06:37:50.524900 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:50.524932 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.124:45855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 06:37:50.525084 7245 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.157.124:45855: Network error: Client connection negotiation failed: client connection to 127.16.157.124:45855: connect: Connection refused (error 111)
W20260928 06:37:50.528409 7327 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 06:37:50.528586 7327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:50.528604 7327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:50.528669 7327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:50.529871 7327 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:37:50.529910 7327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:50.529925 7327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:50.529966 7327 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:37:50.529989 7327 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:37:50.531423 7327 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.157.84:43701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.157.126:40389,127.16.157.125:41087,127.16.157.124:45855
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/krb5kdc/kudu_127.16.157.124.keytab
--principal=kudu/127.16.157.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.157.124:45855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.157.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:37:50.531653 7327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:50.531868 7327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:50.536550 7335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:50.536568 7332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:50.536662 7327 server_base.cc:1034] running on GCE node
W20260928 06:37:50.536571 7333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:50.537027 7327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:50.537302 7327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:50.538443 7327 hybrid_clock.cc:648] HybridClock initialized: now 1790577470538433 us; error 26 us; skew 500 ppm
Sep 28 06:37:50 dist-test-slave-rrj5 krb5kdc[7164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577470, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:50.540984 7327 init.cc:377] Logged in from keytab as kudu/127.16.157.124@KRBTEST.COM (short username kudu)
I20260928 06:37:50.541388 7327 webserver.cc:460] Webserver started at http://127.16.157.124:41221/ using document root <none> and password file <none>
I20260928 06:37:50.541651 7327 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:50.541716 7327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:50.541879 7327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:37:50.542814 7327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/minicluster-data/master-2/data/instance:
uuid: "4f52b4fb7c204f3c9ab7bffc8896c426"
format_stamp: "Formatted at 2026-09-28 06:37:50 on dist-test-slave-rrj5"
server_key: "9d0867a84284525682950b3fc6b9169b"
server_key_iv: "d3b3bf2e2bce8e7901a92d7a4bc39e5c"
server_key_version: "encryptionkey@0"
I20260928 06:37:50.543149 7327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/minicluster-data/master-2/wal/instance:
uuid: "4f52b4fb7c204f3c9ab7bffc8896c426"
format_stamp: "Formatted at 2026-09-28 06:37:50 on dist-test-slave-rrj5"
server_key: "9d0867a84284525682950b3fc6b9169b"
server_key_iv: "d3b3bf2e2bce8e7901a92d7a4bc39e5c"
server_key_version: "encryptionkey@0"
I20260928 06:37:50.544526 7327 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:37:50.545384 7342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:50.545648 7327 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:37:50.545763 7327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/minicluster-data/master-2/data,/tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/minicluster-data/master-2/wal
uuid: "4f52b4fb7c204f3c9ab7bffc8896c426"
format_stamp: "Formatted at 2026-09-28 06:37:50 on dist-test-slave-rrj5"
server_key: "9d0867a84284525682950b3fc6b9169b"
server_key_iv: "d3b3bf2e2bce8e7901a92d7a4bc39e5c"
server_key_version: "encryptionkey@0"
I20260928 06:37:50.545847 7327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:50.557245 7320 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.157.124" port: 45855 } attempt: 2
I20260928 06:37:50.557673 7323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:50.557501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.124:45855 (local address 127.0.0.1:45712)
0928 06:37:50.557561 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:50.557637 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.124:45855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 06:37:50.558014 7323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:50.557929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.124:45855 (local address 127.0.0.1:45716)
0928 06:37:50.557974 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:50.557998 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.157.124:45855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 06:37:50.558130 7320 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.157.124:45855: Network error: Client connection negotiation failed: client connection to 127.16.157.124:45855: connect: Connection refused (error 111)
I20260928 06:37:50.562601 7327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:50.564327 7327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:50.564484 7327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:50.568605 7327 rpc_server.cc:307] RPC server started. Bound to: 127.16.157.124:45855
I20260928 06:37:50.568636 7394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.157.124:45855 every 8 connection(s)
I20260928 06:37:50.568984 7327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/minicluster-data/master-2/data/info.pb
I20260928 06:37:50.569841 17013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGDaHLl/build/release/bin/kudu as pid 7327
I20260928 06:37:50.569927 17013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGDaHLl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577196946045-17013-0/minicluster-data/master-2/wal/instance
I20260928 06:37:50.569943 7395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:37:50.570140 17013 external_mini_cluster.cc:1428] Setting key b7224d8268ae787ca8bf2115ec933cb1
I20260928 06:37:50.572281 7395 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40389 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:37:50 dist-test-slave-rrj5 krb5kdc[7164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.157.124@KRBTEST.COM
Sep 28 06:37:50 dist-test-slave-rrj5 krb5kdc[7164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577470, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.124@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:37:50.580929 7248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:50.573018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60248 (local address 127.16.157.126:40389)
0928 06:37:50.573087 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:50.573089 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:50.573231 (+ 142us) server_negotiation.cc:407] Connection header received
0928 06:37:50.573293 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:50.573294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:50.573302 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:50.573350 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:50.573927 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:50.574663 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:50.575754 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:50.576005 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:50.576079 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:50.578407 (+ 2328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:50.578409 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:50.578409 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:50.578424 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:50.579671 (+ 1247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:50.580057 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:50.580059 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:50.580059 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:50.580119 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:50.580361 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:50.580361 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:50.580362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:50.580477 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:50.580549 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:50.580760 (+ 211us) server_negotiation.cc:299] Negotiation successful
0928 06:37:50.580841 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 06:37:50.581153 7399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:50.572968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.126:40389 (local address 127.0.0.1:60248)
0928 06:37:50.573121 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:50.573161 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:50.573235 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:50.573387 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:50.573389 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:50.573762 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:37:50.573845 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:50.573847 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:50.574760 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:50.574760 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:50.575643 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:50.575645 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:50.576094 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:50.576094 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:50.576153 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:50.576642 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:37:50.576650 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:37:50.578307 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:37:50.579759 (+ 1452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:50.579767 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:50.579769 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:50.579978 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:50.580191 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:50.580191 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:50.580192 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:50.580290 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:50.580554 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:37:50.580561 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:37:50.580636 (+ 75us) client_negotiation.cc:769] Sending connection context
0928 06:37:50.580731 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 06:37:50.580879 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":96,"threads_started":1}
I20260928 06:37:50.581168 7398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:50.570647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45726 (local address 127.16.157.124:45855)
0928 06:37:50.570851 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:50.570861 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:50.570894 (+ 33us) server_negotiation.cc:407] Connection header received
0928 06:37:50.571559 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:50.571568 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:50.571632 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:50.571782 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:50.572379 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:50.573708 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:50.574753 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:50.575370 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:50.575481 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:50.577893 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:50.577901 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:50.577901 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:50.577908 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:50.579707 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:50.580086 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:50.580089 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:50.580089 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:50.580174 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:50.580398 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:50.580399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:50.580399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:50.580605 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:50.580740 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:50.580845 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 06:37:50.580946 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":154,"thread_start_us":91,"threads_started":1}
I20260928 06:37:50.581791 7395 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.157.125" port: 41087 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:37:50 dist-test-slave-rrj5 krb5kdc[7164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577470, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.124@KRBTEST.COM for kudu/127.16.157.125@KRBTEST.COM
Sep 28 06:37:50 dist-test-slave-rrj5 krb5kdc[7164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.157.126@KRBTEST.COM
I20260928 06:37:50.588989 7399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:50.582026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.125:41087 (local address 127.0.0.1:33294)
0928 06:37:50.582070 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:50.582080 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:50.582120 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:50.582304 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:50.582305 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:50.582464 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:37:50.582507 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:50.582509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:50.583241 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:50.583242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:50.583818 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:50.583820 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:50.584303 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:50.584303 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:50.584378 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:50.584861 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:37:50.584866 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:37:50.586066 (+ 1200us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:37:50.587924 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:50.587925 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:50.587925 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:50.588073 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:50.588330 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:50.588331 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:50.588331 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:50.588410 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:50.588640 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:37:50.588641 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:37:50.588686 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 06:37:50.588747 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 06:37:50.588860 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 06:37:50.589058 7324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:50.582140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33294 (local address 127.16.157.125:41087)
0928 06:37:50.582185 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:50.582187 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:50.582195 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:37:50.582228 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:50.582228 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:50.582239 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:50.582294 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:50.582582 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:50.583169 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:50.583894 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:50.584196 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:50.584268 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:50.586148 (+ 1880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:50.586150 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:50.586150 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:50.586157 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:50.587836 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:50.588166 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:50.588167 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:50.588167 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:50.588239 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:50.588473 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:50.588474 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:50.588475 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:50.588577 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:50.588623 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:50.588823 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 06:37:50.588913 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:37:50.589349 7395 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.157.124" port: 45855 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:37:50.589564 7248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:50.581988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60262 (local address 127.16.157.126:40389)
0928 06:37:50.582020 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:50.582022 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:50.582235 (+ 213us) server_negotiation.cc:407] Connection header received
0928 06:37:50.582296 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:50.582297 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:50.582305 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:50.582367 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:50.582893 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:50.583385 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:50.584076 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:50.584305 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:50.584369 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:50.586837 (+ 2468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:50.586838 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:50.586838 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:50.586844 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:50.588325 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:50.588678 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:50.588678 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:50.588679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:50.588755 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:50.588994 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:50.588996 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:50.588996 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:50.589131 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:50.589342 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:50.589382 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 06:37:50.589453 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
Sep 28 06:37:50 dist-test-slave-rrj5 krb5kdc[7164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577470, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.124@KRBTEST.COM for kudu/127.16.157.124@KRBTEST.COM
Sep 28 06:37:50 dist-test-slave-rrj5 krb5kdc[7164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.157.125@KRBTEST.COM
I20260928 06:37:50.595939 7399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:50.589560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.124:45855 (local address 127.0.0.1:45728)
0928 06:37:50.589626 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:50.589638 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:50.589678 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:50.589881 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:50.589882 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:50.590024 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:37:50.590051 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:50.590053 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:50.590639 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:50.590640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:50.591157 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:50.591178 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:50.591497 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:50.591498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:50.591552 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:50.591889 (+ 337us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:37:50.591894 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:37:50.593293 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:37:50.594938 (+ 1645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:50.594939 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:50.594939 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:50.595102 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:50.595317 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:50.595318 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:50.595318 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:50.595369 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:50.595612 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:37:50.595613 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:37:50.595649 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 06:37:50.595718 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 06:37:50.595813 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 06:37:50.596060 7398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:50.589749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45728 (local address 127.16.157.124:45855)
0928 06:37:50.589788 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:50.589789 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:50.589794 (+ 5us) server_negotiation.cc:407] Connection header received
0928 06:37:50.589808 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:50.589809 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:50.589816 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:50.589854 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:50.590126 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:50.590566 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:50.591248 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:50.591431 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:50.591475 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:50.593367 (+ 1892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:50.593368 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:50.593368 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:50.593375 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:50.594858 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:50.595186 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:50.595187 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:50.595187 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:50.595242 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:50.595443 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:50.595444 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:50.595444 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:50.595546 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:50.595619 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:50.595742 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 06:37:50.595864 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":19}
I20260928 06:37:50.597303 7324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:50.590115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33296 (local address 127.16.157.125:41087)
0928 06:37:50.590155 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:50.590156 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:50.590165 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:37:50.590207 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:50.590208 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:50.590217 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:50.590262 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:50.590703 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:50.591132 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:50.591975 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:50.592149 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:50.592200 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:50.594574 (+ 2374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:50.594575 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:50.594575 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:50.594580 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:50.595929 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:50.596630 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:50.596631 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:50.596631 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:50.596688 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:50.596832 (+ 144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:50.596833 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:50.596833 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:50.596944 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:50.597112 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:50.597148 (+ 36us) server_negotiation.cc:299] Negotiation successful
0928 06:37:50.597222 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:37:50.597759 7395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f52b4fb7c204f3c9ab7bffc8896c426: Bootstrap starting.
I20260928 06:37:50.598794 17013 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:37:50.598824 7395 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f52b4fb7c204f3c9ab7bffc8896c426: Neither blocks nor log segments found. Creating new log.
I20260928 06:37:50.599145 7395 log.cc:824] T 00000000000000000000000000000000 P 4f52b4fb7c204f3c9ab7bffc8896c426: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:50.599925 7395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f52b4fb7c204f3c9ab7bffc8896c426: No bootstrap required, opened a new log
I20260928 06:37:50.600250 7245 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.157.124" port: 45855 } attempt: 2
I20260928 06:37:50.599906 7219 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "e15fdbb090ae45729cd66cb9c7bf0ccf"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60262
I20260928 06:37:50.601722 7395 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f52b4fb7c204f3c9ab7bffc8896c426 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e15fdbb090ae45729cd66cb9c7bf0ccf" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40389 } } peers { permanent_uuid: "84431694103a4a939c972cbca69358d7" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 41087 } } peers { permanent_uuid: "4f52b4fb7c204f3c9ab7bffc8896c426" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 45855 } }
I20260928 06:37:50.601854 7395 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f52b4fb7c204f3c9ab7bffc8896c426 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f52b4fb7c204f3c9ab7bffc8896c426, State: Initialized, Role: FOLLOWER
I20260928 06:37:50.601966 7395 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f52b4fb7c204f3c9ab7bffc8896c426 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e15fdbb090ae45729cd66cb9c7bf0ccf" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40389 } } peers { permanent_uuid: "84431694103a4a939c972cbca69358d7" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 41087 } } peers { permanent_uuid: "4f52b4fb7c204f3c9ab7bffc8896c426" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 45855 } }
I20260928 06:37:50.602236 7401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f52b4fb7c204f3c9ab7bffc8896c426 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e15fdbb090ae45729cd66cb9c7bf0ccf" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40389 } } peers { permanent_uuid: "84431694103a4a939c972cbca69358d7" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 41087 } } peers { permanent_uuid: "4f52b4fb7c204f3c9ab7bffc8896c426" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 45855 } } }
I20260928 06:37:50.602304 7395 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f52b4fb7c204f3c9ab7bffc8896c426 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:37:50.602366 7401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f52b4fb7c204f3c9ab7bffc8896c426 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 06:37:50.604234 7412 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4f52b4fb7c204f3c9ab7bffc8896c426: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:37:50.604296 7412 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 28 06:37:50 dist-test-slave-rrj5 krb5kdc[7164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577470, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.126@KRBTEST.COM for kudu/127.16.157.124@KRBTEST.COM
I20260928 06:37:50.604449 7327 master_runner.cc:186] Error getting master registration for 127.16.157.126:40389: OK, instance_id { permanent_uuid: "e15fdbb090ae45729cd66cb9c7bf0ccf" instance_seqno: 1790577470256417 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 06:37:50.607265 7327 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 06:37:50.609318 7398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:50.600546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45742 (local address 127.16.157.124:45855)
0928 06:37:50.602360 (+ 1814us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:50.602363 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:50.602372 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:37:50.602392 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:50.602393 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:50.602402 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:50.602494 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:50.602818 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:50.603584 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:50.604503 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:50.604797 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:50.604857 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:50.606693 (+ 1836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:50.606694 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:50.606695 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:50.606700 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:50.608163 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:50.608527 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:50.608528 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:50.608528 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:50.608594 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:50.608805 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:50.608805 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:50.608806 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:50.608946 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:50.609017 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:50.609168 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 06:37:50.609236 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1797}
I20260928 06:37:50.609333 7247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:50.600490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.124:45855 (local address 127.0.0.1:45742)
0928 06:37:50.600526 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:50.600537 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:50.600574 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:50.602523 (+ 1949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:50.602524 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:50.602681 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:37:50.602720 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:50.602723 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:50.603648 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:50.603661 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:50.604416 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:50.604419 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:50.604857 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:50.604858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:50.604926 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:50.605321 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:37:50.605326 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:37:50.606609 (+ 1283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:37:50.608267 (+ 1658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:50.608268 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:50.608268 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:50.608440 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:50.608666 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:50.608666 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:50.608667 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:50.608733 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:50.609011 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:37:50.609011 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:37:50.609056 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 06:37:50.609143 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 06:37:50.609236 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260928 06:37:50.610493 7245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e15fdbb090ae45729cd66cb9c7bf0ccf: Bootstrap starting.
I20260928 06:37:50.611030 7245 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e15fdbb090ae45729cd66cb9c7bf0ccf: Neither blocks nor log segments found. Creating new log.
I20260928 06:37:50.611264 7245 log.cc:824] T 00000000000000000000000000000000 P e15fdbb090ae45729cd66cb9c7bf0ccf: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:50.611925 7245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e15fdbb090ae45729cd66cb9c7bf0ccf: No bootstrap required, opened a new log
I20260928 06:37:50.612823 7245 raft_consensus.cc:348] T 00000000000000000000000000000000 P e15fdbb090ae45729cd66cb9c7bf0ccf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e15fdbb090ae45729cd66cb9c7bf0ccf" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40389 } } peers { permanent_uuid: "84431694103a4a939c972cbca69358d7" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 41087 } } peers { permanent_uuid: "4f52b4fb7c204f3c9ab7bffc8896c426" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 45855 } }
I20260928 06:37:50.612921 7245 raft_consensus.cc:729] T 00000000000000000000000000000000 P e15fdbb090ae45729cd66cb9c7bf0ccf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e15fdbb090ae45729cd66cb9c7bf0ccf, State: Initialized, Role: FOLLOWER
I20260928 06:37:50.612977 7245 consensus_queue.cc:260] T 00000000000000000000000000000000 P e15fdbb090ae45729cd66cb9c7bf0ccf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e15fdbb090ae45729cd66cb9c7bf0ccf" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40389 } } peers { permanent_uuid: "84431694103a4a939c972cbca69358d7" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 41087 } } peers { permanent_uuid: "4f52b4fb7c204f3c9ab7bffc8896c426" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 45855 } }
I20260928 06:37:50.613217 7415 sys_catalog.cc:455] T 00000000000000000000000000000000 P e15fdbb090ae45729cd66cb9c7bf0ccf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e15fdbb090ae45729cd66cb9c7bf0ccf" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40389 } } peers { permanent_uuid: "84431694103a4a939c972cbca69358d7" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 41087 } } peers { permanent_uuid: "4f52b4fb7c204f3c9ab7bffc8896c426" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 45855 } } }
I20260928 06:37:50.613271 7245 sys_catalog.cc:564] T 00000000000000000000000000000000 P e15fdbb090ae45729cd66cb9c7bf0ccf [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:37:50.613317 7219 raft_consensus.cc:482] T 00000000000000000000000000000000 P e15fdbb090ae45729cd66cb9c7bf0ccf [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 06:37:50.613317 7415 sys_catalog.cc:458] T 00000000000000000000000000000000 P e15fdbb090ae45729cd66cb9c7bf0ccf [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:37:50.613418 7219 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e15fdbb090ae45729cd66cb9c7bf0ccf [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:37:50.614415 7219 raft_consensus.cc:504] T 00000000000000000000000000000000 P e15fdbb090ae45729cd66cb9c7bf0ccf [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e15fdbb090ae45729cd66cb9c7bf0ccf" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40389 } } peers { permanent_uuid: "84431694103a4a939c972cbca69358d7" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 41087 } } peers { permanent_uuid: "4f52b4fb7c204f3c9ab7bffc8896c426" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 45855 } }
W20260928 06:37:50.615486 7426 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e15fdbb090ae45729cd66cb9c7bf0ccf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:37:50.615548 7426 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:37:50.614661 7219 leader_election.cc:288] T 00000000000000000000000000000000 P e15fdbb090ae45729cd66cb9c7bf0ccf [CANDIDATE]: Term 1 election: Requested vote from peers 84431694103a4a939c972cbca69358d7 (127.16.157.125:41087), 4f52b4fb7c204f3c9ab7bffc8896c426 (127.16.157.124:45855)
I20260928 06:37:50.615914 7369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e15fdbb090ae45729cd66cb9c7bf0ccf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4f52b4fb7c204f3c9ab7bffc8896c426"
I20260928 06:37:50.616012 7369 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f52b4fb7c204f3c9ab7bffc8896c426 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:37:50.616914 7369 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4f52b4fb7c204f3c9ab7bffc8896c426 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e15fdbb090ae45729cd66cb9c7bf0ccf in term 1.
I20260928 06:37:50.617153 7195 leader_election.cc:302] T 00000000000000000000000000000000 P e15fdbb090ae45729cd66cb9c7bf0ccf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f52b4fb7c204f3c9ab7bffc8896c426, e15fdbb090ae45729cd66cb9c7bf0ccf; no voters:
I20260928 06:37:50.617287 7415 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e15fdbb090ae45729cd66cb9c7bf0ccf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:37:50.617455 7415 raft_consensus.cc:686] T 00000000000000000000000000000000 P e15fdbb090ae45729cd66cb9c7bf0ccf [term 1 LEADER]: Becoming Leader. State: Replica: e15fdbb090ae45729cd66cb9c7bf0ccf, State: Running, Role: LEADER
I20260928 06:37:50.617542 7415 consensus_queue.cc:237] T 00000000000000000000000000000000 P e15fdbb090ae45729cd66cb9c7bf0ccf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e15fdbb090ae45729cd66cb9c7bf0ccf" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40389 } } peers { permanent_uuid: "84431694103a4a939c972cbca69358d7" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 41087 } } peers { permanent_uuid: "4f52b4fb7c204f3c9ab7bffc8896c426" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 45855 } }
I20260928 06:37:50.617980 7428 sys_catalog.cc:455] T 00000000000000000000000000000000 P e15fdbb090ae45729cd66cb9c7bf0ccf [sys.catalog]: SysCatalogTable state changed. Reason: New leader e15fdbb090ae45729cd66cb9c7bf0ccf. Latest consensus state: current_term: 1 leader_uuid: "e15fdbb090ae45729cd66cb9c7bf0ccf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e15fdbb090ae45729cd66cb9c7bf0ccf" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40389 } } peers { permanent_uuid: "84431694103a4a939c972cbca69358d7" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 41087 } } peers { permanent_uuid: "4f52b4fb7c204f3c9ab7bffc8896c426" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 45855 } } }
I20260928 06:37:50.615841 7177 master_runner.cc:186] Error getting master registration for 127.16.157.125:41087: OK, instance_id { permanent_uuid: "84431694103a4a939c972cbca69358d7" instance_seqno: 1790577470413910 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 06:37:50.618261 7177 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 06:37:50.618193 7428 sys_catalog.cc:458] T 00000000000000000000000000000000 P e15fdbb090ae45729cd66cb9c7bf0ccf [sys.catalog]: This master's current role is: LEADER
I20260928 06:37:50.621721 7430 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:37:50.621903 7430 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:37:50.624270 7369 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4f52b4fb7c204f3c9ab7bffc8896c426 [term 1 FOLLOWER]: Refusing update from remote peer e15fdbb090ae45729cd66cb9c7bf0ccf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:37:50.624727 7415 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e15fdbb090ae45729cd66cb9c7bf0ccf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f52b4fb7c204f3c9ab7bffc8896c426" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 45855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:37:50.626276 7401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f52b4fb7c204f3c9ab7bffc8896c426 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e15fdbb090ae45729cd66cb9c7bf0ccf. Latest consensus state: current_term: 1 leader_uuid: "e15fdbb090ae45729cd66cb9c7bf0ccf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e15fdbb090ae45729cd66cb9c7bf0ccf" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40389 } } peers { permanent_uuid: "84431694103a4a939c972cbca69358d7" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 41087 } } peers { permanent_uuid: "4f52b4fb7c204f3c9ab7bffc8896c426" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 45855 } } }
I20260928 06:37:50.626370 7401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f52b4fb7c204f3c9ab7bffc8896c426 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:37:50.632230 7248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:50.623243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60272 (local address 127.16.157.126:40389)
0928 06:37:50.623276 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:50.623278 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:50.624019 (+ 741us) server_negotiation.cc:407] Connection header received
0928 06:37:50.624111 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:50.624112 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:50.624121 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:50.624149 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:50.624939 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:50.625677 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:50.626720 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:50.627003 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:50.627044 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:50.628907 (+ 1863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:50.628908 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:50.628909 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:50.628916 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:50.630653 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:50.631311 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:50.631312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:50.631313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:50.631385 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:50.631600 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:50.631601 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:50.631601 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:50.631719 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:50.631821 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:50.632068 (+ 247us) server_negotiation.cc:299] Negotiation successful
0928 06:37:50.632142 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 06:37:50.636531 7324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:50.622845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33300 (local address 127.16.157.125:41087)
0928 06:37:50.622885 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:50.622887 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:50.623534 (+ 647us) server_negotiation.cc:407] Connection header received
0928 06:37:50.623692 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:50.623693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:50.623714 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:50.623744 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:50.627835 (+ 4091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:50.628548 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:50.630251 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:50.630552 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:50.630597 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:50.633142 (+ 2545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:50.633143 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:50.633143 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:50.633150 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:50.634813 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:50.635451 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:50.635452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:50.635452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:50.635510 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:50.635837 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:50.635838 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:50.635838 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:50.635966 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:50.636035 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:50.636358 (+ 323us) server_negotiation.cc:299] Negotiation successful
0928 06:37:50.636437 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:37:50.640126 7398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:50.623122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45752 (local address 127.16.157.124:45855)
0928 06:37:50.623167 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:50.623169 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:50.627657 (+ 4488us) server_negotiation.cc:407] Connection header received
0928 06:37:50.627726 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:50.627727 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:50.627737 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:50.627801 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:50.632733 (+ 4932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:50.633781 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:50.635039 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:50.635345 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:50.635387 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:50.636765 (+ 1378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:50.636766 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:50.636767 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:50.636774 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:50.638558 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:50.639268 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:50.639269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:50.639270 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:50.639332 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:50.639553 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:50.639555 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:50.639555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:50.639701 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:50.639807 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:50.639940 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 06:37:50.640033 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:37:50.642220 7415 sys_catalog.cc:455] T 00000000000000000000000000000000 P e15fdbb090ae45729cd66cb9c7bf0ccf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e15fdbb090ae45729cd66cb9c7bf0ccf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e15fdbb090ae45729cd66cb9c7bf0ccf" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40389 } } peers { permanent_uuid: "84431694103a4a939c972cbca69358d7" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 41087 } } peers { permanent_uuid: "4f52b4fb7c204f3c9ab7bffc8896c426" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 45855 } } }
I20260928 06:37:50.642323 7415 sys_catalog.cc:458] T 00000000000000000000000000000000 P e15fdbb090ae45729cd66cb9c7bf0ccf [sys.catalog]: This master's current role is: LEADER
I20260928 06:37:50.642390 7401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f52b4fb7c204f3c9ab7bffc8896c426 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e15fdbb090ae45729cd66cb9c7bf0ccf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e15fdbb090ae45729cd66cb9c7bf0ccf" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40389 } } peers { permanent_uuid: "84431694103a4a939c972cbca69358d7" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 41087 } } peers { permanent_uuid: "4f52b4fb7c204f3c9ab7bffc8896c426" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 45855 } } }
I20260928 06:37:50.642479 7401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f52b4fb7c204f3c9ab7bffc8896c426 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:37:50.642658 7435 mvcc.cc:205] Tried to move back new op lower bound from 7334205319674318848 to 7334205319650201600. Current Snapshot: MvccSnapshot[applied={T|T < 7334205319674318848}]
I20260928 06:37:50.646193 7430 catalog_manager.cc:1347] Generated new cluster ID: f819c8e1a254429da3300b4559df3a93
I20260928 06:37:50.646260 7430 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:37:50.649214 7320 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.157.124" port: 45855 } attempt: 3
Sep 28 06:37:50 dist-test-slave-rrj5 krb5kdc[7164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577470, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.157.125@KRBTEST.COM for kudu/127.16.157.124@KRBTEST.COM
I20260928 06:37:50.662397 7323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:50.649459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.157.124:45855 (local address 127.0.0.1:45764)
0928 06:37:50.649548 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:50.649558 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:50.649618 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:50.653884 (+ 4266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:50.653885 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:50.654089 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:37:50.654139 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:50.654142 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:50.655098 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:50.655099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:50.655955 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:50.655957 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:50.656508 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:50.656509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:50.656584 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:50.657198 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:37:50.657204 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:37:50.658876 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:37:50.660984 (+ 2108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:50.660985 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:50.660985 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:50.661225 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:50.661519 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:50.661520 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:50.661521 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:50.661607 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:50.662086 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:37:50.662087 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:37:50.662145 (+ 58us) client_negotiation.cc:769] Sending connection context
0928 06:37:50.662193 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 06:37:50.662293 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260928 06:37:50.663272 7398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:50.653642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45764 (local address 127.16.157.124:45855)
0928 06:37:50.653707 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:50.653709 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:50.653718 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:37:50.653754 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:50.653754 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:50.653764 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:50.653828 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:50.654242 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:50.655001 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:50.656068 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:50.656379 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:50.656436 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:50.658988 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:50.658989 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:50.658990 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:50.658997 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:50.660871 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:50.661316 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:50.661317 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:50.661317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:50.661404 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:50.661695 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:50.661695 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:50.661696 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:50.661953 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:50.662858 (+ 905us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:50.662979 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 06:37:50.663120 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 06:37:50.665112 7320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84431694103a4a939c972cbca69358d7: Bootstrap starting.
I20260928 06:37:50.665931 7320 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84431694103a4a939c972cbca69358d7: Neither blocks nor log segments found. Creating new log.
I20260928 06:37:50.666236 7320 log.cc:824] T 00000000000000000000000000000000 P 84431694103a4a939c972cbca69358d7: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:50.670209 7320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84431694103a4a939c972cbca69358d7: No bootstrap required, opened a new log
I20260928 06:37:50.672338 7320 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84431694103a4a939c972cbca69358d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e15fdbb090ae45729cd66cb9c7bf0ccf" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40389 } } peers { permanent_uuid: "84431694103a4a939c972cbca69358d7" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 41087 } } peers { permanent_uuid: "4f52b4fb7c204f3c9ab7bffc8896c426" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 45855 } }
I20260928 06:37:50.672506 7320 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84431694103a4a939c972cbca69358d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84431694103a4a939c972cbca69358d7, State: Initialized, Role: FOLLOWER
I20260928 06:37:50.672731 7320 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84431694103a4a939c972cbca69358d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e15fdbb090ae45729cd66cb9c7bf0ccf" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40389 } } peers { permanent_uuid: "84431694103a4a939c972cbca69358d7" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 41087 } } peers { permanent_uuid: "4f52b4fb7c204f3c9ab7bffc8896c426" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 45855 } }
I20260928 06:37:50.673127 7320 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84431694103a4a939c972cbca69358d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:37:50.673185 7294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e15fdbb090ae45729cd66cb9c7bf0ccf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "84431694103a4a939c972cbca69358d7"
I20260928 06:37:50.673288 7294 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84431694103a4a939c972cbca69358d7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:37:50.674010 7430 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:37:50.674606 7294 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 84431694103a4a939c972cbca69358d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e15fdbb090ae45729cd66cb9c7bf0ccf in term 1.
I20260928 06:37:50.674786 7430 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:37:50.675086 7293 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 84431694103a4a939c972cbca69358d7 [term 1 FOLLOWER]: Refusing update from remote peer e15fdbb090ae45729cd66cb9c7bf0ccf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:37:50.675706 7415 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e15fdbb090ae45729cd66cb9c7bf0ccf [LEADER]: Connected to new peer: Peer: permanent_uuid: "84431694103a4a939c972cbca69358d7" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 41087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:37:50.677393 7444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84431694103a4a939c972cbca69358d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e15fdbb090ae45729cd66cb9c7bf0ccf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e15fdbb090ae45729cd66cb9c7bf0ccf" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40389 } } peers { permanent_uuid: "84431694103a4a939c972cbca69358d7" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 41087 } } peers { permanent_uuid: "4f52b4fb7c204f3c9ab7bffc8896c426" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 45855 } } }
I20260928 06:37:50.677615 7444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84431694103a4a939c972cbca69358d7 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:37:50.677748 7444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84431694103a4a939c972cbca69358d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e15fdbb090ae45729cd66cb9c7bf0ccf. Latest consensus state: current_term: 1 leader_uuid: "e15fdbb090ae45729cd66cb9c7bf0ccf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e15fdbb090ae45729cd66cb9c7bf0ccf" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40389 } } peers { permanent_uuid: "84431694103a4a939c972cbca69358d7" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 41087 } } peers { permanent_uuid: "4f52b4fb7c204f3c9ab7bffc8896c426" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 45855 } } }
I20260928 06:37:50.677865 7444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84431694103a4a939c972cbca69358d7 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:37:50.677951 7444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84431694103a4a939c972cbca69358d7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e15fdbb090ae45729cd66cb9c7bf0ccf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e15fdbb090ae45729cd66cb9c7bf0ccf" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40389 } } peers { permanent_uuid: "84431694103a4a939c972cbca69358d7" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 41087 } } peers { permanent_uuid: "4f52b4fb7c204f3c9ab7bffc8896c426" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 45855 } } }
I20260928 06:37:50.678056 7444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84431694103a4a939c972cbca69358d7 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:37:50.678897 7430 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e15fdbb090ae45729cd66cb9c7bf0ccf: Generated new TSK 0
I20260928 06:37:50.679090 7430 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 06:37:50.680610 7459 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 84431694103a4a939c972cbca69358d7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:37:50.680678 7459 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:37:50.681102 7415 sys_catalog.cc:455] T 00000000000000000000000000000000 P e15fdbb090ae45729cd66cb9c7bf0ccf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e15fdbb090ae45729cd66cb9c7bf0ccf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e15fdbb090ae45729cd66cb9c7bf0ccf" member_type: VOTER last_known_addr { host: "127.16.157.126" port: 40389 } } peers { permanent_uuid: "84431694103a4a939c972cbca69358d7" member_type: VOTER last_known_addr { host: "127.16.157.125" port: 41087 } } peers { permanent_uuid: "4f52b4fb7c204f3c9ab7bffc8896c426" member_type: VOTER last_known_addr { host: "127.16.157.124" port: 45855 } } }
I20260928 06:37:50.681195 7415 sys_catalog.cc:458] T 00000000000000000000000000000000 P e15fdbb090ae45729cd66cb9c7bf0ccf [sys.catalog]: This master's current role is: LEADER
I20260928 06:37:50.681717 7449 mvcc.cc:205] Tried to move back new op lower bound from 7334205319872495616 to 7334205319650201600. Current Snapshot: MvccSnapshot[applied={T|T < 7334205319674318848}]
W20260928 06:37:50.712802 7324 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:37:50.711003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33314 (local address 127.16.157.125:41087)
0928 06:37:50.711055 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:50.711057 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:50.712087 (+ 1030us) server_negotiation.cc:407] Connection header received
0928 06:37:50.712195 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:50.712196 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:50.712208 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:50.712239 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:50.712717 (+ 478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33314: BlockingRecv error: recv got EOF from 127.0.0.1:33314 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260928 06:37:50.713557 7398 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:37:50.711456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45778 (local address 127.16.157.124:45855)
0928 06:37:50.711496 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:50.711498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:50.713061 (+ 1563us) server_negotiation.cc:407] Connection header received
0928 06:37:50.713160 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:50.713161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:50.713172 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:50.713200 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:50.713496 (+ 296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45778: BlockingRecv error: recv got EOF from 127.0.0.1:45778 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 06:37:50.714306 7248 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:37:50.711679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60282 (local address 127.16.157.126:40389)
0928 06:37:50.711722 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:50.711723 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:50.713784 (+ 2061us) server_negotiation.cc:407] Connection header received
0928 06:37:50.713900 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:50.713901 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:50.713912 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:50.713939 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:50.714248 (+ 309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60282: BlockingRecv error: recv got EOF from 127.0.0.1:60282 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 06:37:50.716806 7324 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:37:50.715973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33330 (local address 127.16.157.125:41087)
0928 06:37:50.716010 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:50.716011 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:50.716244 (+ 233us) server_negotiation.cc:407] Connection header received
0928 06:37:50.716358 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:50.716359 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:50.716377 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:50.716409 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:50.716763 (+ 354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33330: BlockingRecv error: recv got EOF from 127.0.0.1:33330 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 06:37:50.722709 7324 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:37:50.721997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33336 (local address 127.16.157.125:41087)
0928 06:37:50.722060 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:50.722062 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:50.722196 (+ 134us) server_negotiation.cc:407] Connection header received
0928 06:37:50.722296 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:50.722297 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:50.722309 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:50.722338 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:50.722663 (+ 325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33336: BlockingRecv error: recv got EOF from 127.0.0.1:33336 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260928 06:37:50.726851 7324 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:37:50.725989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33352 (local address 127.16.157.125:41087)
0928 06:37:50.726030 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:50.726032 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:50.726318 (+ 286us) server_negotiation.cc:407] Connection header received
0928 06:37:50.726417 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:50.726418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:50.726431 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:50.726460 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:50.726802 (+ 342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33352: BlockingRecv error: recv got EOF from 127.0.0.1:33352 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 06:37:50.733114 7324 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:37:50.732161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33356 (local address 127.16.157.125:41087)
0928 06:37:50.732202 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:50.732204 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:50.732497 (+ 293us) server_negotiation.cc:407] Connection header received
0928 06:37:50.732601 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:50.732602 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:50.732613 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:50.732643 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:50.733057 (+ 414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33356: BlockingRecv error: recv got EOF from 127.0.0.1:33356 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 06:37:50.743405 7324 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:37:50.742462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33370 (local address 127.16.157.125:41087)
0928 06:37:50.742506 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:50.742508 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:50.742786 (+ 278us) server_negotiation.cc:407] Connection header received
0928 06:37:50.742899 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:50.742900 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:50.742912 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:50.742963 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:50.743347 (+ 384us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33370: BlockingRecv error: recv got EOF from 127.0.0.1:33370 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 06:37:50.761965 7324 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:37:50.760882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33372 (local address 127.16.157.125:41087)
0928 06:37:50.760929 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:50.760931 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:50.761245 (+ 314us) server_negotiation.cc:407] Connection header received
0928 06:37:50.761376 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:50.761377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:50.761392 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:50.761430 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:50.761903 (+ 473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33372: BlockingRecv error: recv got EOF from 127.0.0.1:33372 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 06:37:50.796197 7324 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:37:50.795385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33374 (local address 127.16.157.125:41087)
0928 06:37:50.795437 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:50.795440 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:50.795610 (+ 170us) server_negotiation.cc:407] Connection header received
0928 06:37:50.795684 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:50.795685 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:50.795697 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:50.795748 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:50.796121 (+ 373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33374: BlockingRecv error: recv got EOF from 127.0.0.1:33374 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260928 06:37:50.862282 7324 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:37:50.861556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33382 (local address 127.16.157.125:41087)
0928 06:37:50.861674 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:50.861676 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:50.861703 (+ 27us) server_negotiation.cc:407] Connection header received
0928 06:37:50.861782 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:50.861783 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:50.861793 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:50.861843 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:50.862204 (+ 361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33382: BlockingRecv error: recv got EOF from 127.0.0.1:33382 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260928 06:37:50.992141 7324 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:37:50.991480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33398 (local address 127.16.157.125:41087)
0928 06:37:50.991555 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:50.991557 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:50.991658 (+ 101us) server_negotiation.cc:407] Connection header received
0928 06:37:50.991704 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:50.991705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:50.991715 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:50.991787 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:50.992094 (+ 307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33398: BlockingRecv error: recv got EOF from 127.0.0.1:33398 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260928 06:37:51.250188 7324 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:37:51.249466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33402 (local address 127.16.157.125:41087)
0928 06:37:51.249539 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:51.249541 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:51.249599 (+ 58us) server_negotiation.cc:407] Connection header received
0928 06:37:51.249675 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:51.249676 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:51.249686 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:51.249733 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:51.250081 (+ 348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33402: BlockingRecv error: recv got EOF from 127.0.0.1:33402 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 06:37:51.604723 7412 catalog_manager.cc:1259] Loaded cluster ID: f819c8e1a254429da3300b4559df3a93
I20260928 06:37:51.604825 7412 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4f52b4fb7c204f3c9ab7bffc8896c426: loading cluster ID for follower catalog manager: success
I20260928 06:37:51.605724 7412 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4f52b4fb7c204f3c9ab7bffc8896c426: acquiring CA information for follower catalog manager: success
I20260928 06:37:51.606043 7412 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4f52b4fb7c204f3c9ab7bffc8896c426: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 06:37:51.681074 7459 catalog_manager.cc:1259] Loaded cluster ID: f819c8e1a254429da3300b4559df3a93
I20260928 06:37:51.681151 7459 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 84431694103a4a939c972cbca69358d7: loading cluster ID for follower catalog manager: success
I20260928 06:37:51.682066 7459 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 84431694103a4a939c972cbca69358d7: acquiring CA information for follower catalog manager: success
I20260928 06:37:51.682372 7459 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 84431694103a4a939c972cbca69358d7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 06:37:51.765918 7530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:51.763336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33416 (local address 127.16.157.125:41087)
0928 06:37:51.763456 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:51.763458 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:51.763470 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:37:51.763528 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:51.763528 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:51.763578 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:51.763651 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:37:51.763812 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:51.764365 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:51.765133 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:51.765322 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:51.765378 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:51.765517 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:37:51.765672 (+ 155us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:37:51.765733 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:51.765783 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 06:37:51.765802 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":73,"threads_started":1}
I20260928 06:37:51.770151 7536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:51.767826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45790 (local address 127.16.157.124:45855)
0928 06:37:51.767947 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:51.767949 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:51.767958 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:37:51.768003 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:51.768003 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:51.768038 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:51.768106 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:37:51.768200 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:51.768666 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:51.769335 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:51.769518 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:51.769565 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:51.769807 (+ 242us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:37:51.769915 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:37:51.769972 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:51.770012 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 06:37:51.770028 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":78,"threads_started":1}
I20260928 06:37:51.774828 7536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:51.771869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45802 (local address 127.16.157.124:45855)
0928 06:37:51.771912 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:51.771914 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:51.772031 (+ 117us) server_negotiation.cc:407] Connection header received
0928 06:37:51.772116 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:51.772116 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:51.772125 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:51.772176 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:37:51.772304 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:51.773024 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:51.774050 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:51.774352 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:51.774413 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:51.774561 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:37:51.774594 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:37:51.774664 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:51.774728 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 06:37:51.774751 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:37:51.774950 7544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:51.771723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60292 (local address 127.16.157.126:40389)
0928 06:37:51.771828 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:51.771830 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:51.771840 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:37:51.771984 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:51.771985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:51.772036 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:51.772115 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:37:51.772387 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:51.773139 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:51.773961 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:51.774260 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:51.774323 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:51.774512 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:37:51.774602 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:37:51.774659 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:51.774830 (+ 171us) server_negotiation.cc:299] Negotiation successful
0928 06:37:51.774854 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":61,"threads_started":1}
I20260928 06:37:51.775278 7530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:51.771825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33432 (local address 127.16.157.125:41087)
0928 06:37:51.771860 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:51.771862 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:51.772017 (+ 155us) server_negotiation.cc:407] Connection header received
0928 06:37:51.772052 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:51.772053 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:51.772063 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:51.772114 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:37:51.772350 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:51.772899 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:51.774466 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:51.774756 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:51.774814 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:51.775010 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:37:51.775044 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:37:51.775074 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:51.775171 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 06:37:51.775191 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:37:51.779982 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 7177
I20260928 06:37:51.785868 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 7252
I20260928 06:37:51.790921 17013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGDaHLl/build/release/bin/kudu with pid 7327
2026-09-28T06:37:51Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2787 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5585 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (274853 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST